Repository navigation
fix(examine): enumerate Instinct GPUs from lspci's accelerator class - #434
Conversation
…ator class `probe_gpus_lspci` matched only the three display classes, but Instinct parts carry no display class at all: an MI300X enumerates under PCI class 1200, "Processing accelerators". Every datacenter GPU was therefore invisible to the lspci probe, leaving `has_amd_gpu` to rest entirely on `rocminfo` or the sysfs fallback -- and report false when neither was reachable, which is the reported symptom in EAI-8449. Verified on an 8-GPU MI300X host, where all eight devices read `class=0x120000` in sysfs and `lspci -nn -D` renders them as `Processing accelerators [1200]`. Before: one topology-sourced entry with no PCI id and `has_discrete_amd: false`. After: eight entries carrying their real PCI ids, correctly classified as discrete. Picking the richer lspci answer would otherwise have cost the gfx target, since lspci does not know one and `rocminfo` was absent: the entries went to an empty `gfx_target` where the fallback had supplied `gfx942`, and `gpus[].gfx_target` is the only place the JSON report names a target at all. So the fallback now also fills that gap from the kernel topology, under the same rule it already followed for enumeration -- fill a gap, never overwrite a target `rocminfo` resolved per-device. Note this does not change the original report's root cause as filed: the `amdkfd` module gating described there does not exist in the code, and the MI300X host confirms it, reporting `amdkfd_loaded: false` alongside `has_amd_gpu: true`. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Review of the previous commit found the gap-fill it added was unsafe on a mixed host. `detect_kfd_gfx_target_in` deliberately returns the lowest-numbered KFD node -- the right answer for HIP ordinal 0, and the wrong one to pin on a *particular* device. On an APU+dGPU box without `rocminfo`, `lspci` resolves the integrated part by name but leaves the discrete card empty, so the fill stamped the APU's target onto the dGPU. That is not inert: `diagnose`'s iGPU/dGPU check splits on `gfx_target` to tell the user which GPU to pin, and the comment above it says naming the wrong one is "actively wrong for exactly the APU+dGPU pairing this check fires on". Before the fill the discrete entry was empty and the check took its honest branch; after, it named a confident wrong target. So read the topology differently for this purpose: `uniform_kfd_gfx_target_in` reports a target only when every KFD GPU node agrees on one, together with how many nodes it saw, and the fill additionally requires that count to equal the number of AMD GPUs enumerated. A topology that does not account for every card stays silent rather than guessing. The MI300X case that motivated the fill is unaffected -- eight nodes, one target, eight GPUs -- and is re-verified on that hardware. Also from review: - Add the e2e scenario AGENTS.md requires for observable behavior. The two scenarios the previous commit relied on do not cover it: examine-04 and examine-08 assert `detected_gfx_target` and `has_amd_gpu`, neither of which the enumeration change moves, so both passed before and after. examine-16 reads `gpus[]` itself, cross-checked against the kernel's GPU node count, and no-ops where the premise cannot hold. - Correct a doc comment claiming `gpus[].gfx_target` is the only place the JSON names a target; `framework_arch_list` and `summary` carry targets too. The true claim is narrower: the hardware's own target per device. - Describe the fallback's second responsibility in its doc comment, which still said the function only discovers GPUs when nothing else could. - Stop two comments in the same commit contradicting each other about whether lspci names an Instinct part "Device" -- it depends on the host's `pci.ids` vintage. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
🔴 Automated review · pr-review-watcher · 6699ec3 This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer. SummaryTwo commits teach the bus-listing probe to enumerate datacenter GPUs that carry a processing-accelerator class instead of a display class, and add a narrowly-guarded fill that copies a gfx target from the kernel topology onto enumerated GPUs only when the topology accounts for every one of them. Outcome: Needs work — the enumeration half is well-built and well-tested, but the gap-fill half has no test at any level. Verified: the merge base equals the base tip and both diffs produce an identical four-file set; the four new unit tests compile and pass locally (4 passed, 0 failed); I confirmed by reading source that the sorted node ordering makes the topology test deterministic rather than flaky, that non-AMD accelerators matched by the new class rule are dropped by the existing vendor gate so they cannot corrupt the counts, that the count guard can only ever suppress a fill and never produce a wrong one, and that the commit's claim about the two pre-existing scenarios not covering the enumeration change is accurate; a leak scan over the diff returned no hits and no prompt-injection attempt was found in code, comments, commit messages or feature text. The full suite, clippy and the e2e lanes were not run here. This review worked from a CI rollup of 23 success, 1 skipped, 4 failure; I could not attribute any red check to a specific defect — the one candidate is the new scenario's strict count equality (non-blocking item 3), which can fail on a legitimate host, but that is inference and I cannot confirm it. Blocking: 1 · Non-blocking: 5. 🚫 Blocking (must fix before merge)crates/rocm-core/src/examine.rs:1256-1258 — the gfx-target gap-fill, which is the entire second half of this PR, is wired in by a single statement that no test exercises in a state where it can do anything. Delete that line and the whole suite still passes. The two new unit tests call the inner helper directly with hand-supplied arguments, so they prove the helper and nothing about the wiring; the one pre-existing test that now reaches this branch ( Non-blocking
|
siloteemu
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 6699ec3
Blocking: 1. This is the first round of review from this account on this pull request, and it answers the review request.
crates/rocm-core/src/examine.rs:1256-1258 - the gap-fill half of this change is wired in by a single statement that no test exercises in a state where it can do anything.
Delete that statement and the whole suite still passes. The three things that look like coverage are not:
- The two new unit tests call the inner helper directly with hand-supplied arguments, so they prove the helper and say nothing about the wiring.
- The one pre-existing test that now reaches this branch arrives with a GPU whose target is already populated, so the fill is a no-op there and its assertions cannot move either way.
- The new scenario asserts entry count and bus address, but never reads the target field the fill exists to populate.
That field is user-observable output of the machine-readable report, and the contributor rules on the base branch are explicit that a unit test on an internal helper does not discharge the scenario requirement - it proves the function, not the behaviour. So the entire second half of this change could be dropped in a later refactor with every gate green.
The enumeration half is a different matter and is well built: the class-based selection is covered, the vendor gate keeps non-AMD accelerators out of the counts, the node ordering makes the topology read deterministic rather than flaky, and the count guard can only ever suppress a fill, never produce a wrong one. This objection is only about the fill being unpinned.
Concrete fix: extend the new step, or add one, to assert that every AMD entry carries a non-empty target whenever the kernel topology is uniform and its GPU-node count equals the number of AMD entries reported - no-opping on the same premise-failure conditions the step already uses. And give the fill a seam so a unit test can drive the early-return branch against a planted topology instead of the host's, the way the topology reader itself is already split.
One related observation rather than part of the objection: four checks are red at this head. I could not attribute any of them to a defect I found. The one candidate is the new scenario's strict count equality, which can fail on a legitimate host - see the non-blocking items - but that is inference and cannot be confirmed from here.
The full round of findings, including five non-blocking items, is in the review comment posted alongside this one.
juhovainio
left a comment
There was a problem hiding this comment.
I read through the lspci class widening and the KFD gap-fill logic, checked out this commit locally, and ran the unit tests plus clippy against it. The core fix is solid: is_lspci_gpu_line now correctly picks up class 1200 ("Processing accelerators") devices, and uniform_kfd_gfx_target_in/fill_missing_gfx_targets only fill a missing gfx_target when every KFD node agrees and the node count matches the GPU count exactly, refusing otherwise rather than guessing. I hand-checked the GC-version decode arithmetic in the new test fixtures (90402 -> gfx942, 110003/110000 -> gfx1103/gfx1100 correctly refusing to fill) and it's right. All existing and new unit tests pass, clippy is clean.
But this PR's own new e2e scenario (examine-16) is failing on real hardware in this repo's CI, and it looks like a real problem rather than test flakiness:
- MI300X: "the kernel describes 1 GPU node(s) but the report lists 8 AMD GPU(s)" (https://github.com/ROCm/rocm-cli/actions/runs/35969206453/job/107534834932)
- MI350P: same pattern, 1 node vs 2 AMD GPUs (https://github.com/ROCm/rocm-cli/actions/runs/35969206453/job/107534834944)
- rad3 R9700: same pattern, 1 node vs 8 AMD GPUs (https://github.com/ROCm/rocm-cli/actions/runs/35969206453/job/107534834806)
(Strix Halo Ubuntu also failed, but that's the same pre-existing VRAM-preflight infra issue seen on other PRs in this queue, unrelated to this change.)
Looking at the MI300X run in detail: lspci correctly enumerates all 8 physical Instinct GPUs with real, distinct PCI addresses, so the fix itself works as intended. But the same run's JSON shows only one render device (/dev/dri/renderD152) and the KFD topology exposes only one node with a nonzero gfx_target_version. That points to these being shared bare-metal runners: the host has 8 MI300X GPUs, lspci walks the whole PCI bus and sees all of them, but this particular job is only cgroup-granted access to one via KFD/render-node restrictions.
That's more than a flaky test. Before this PR, class-1200 devices weren't matched at all, so a host like this fell through to the old KFD-based fallback, which is scoped to what's actually accessible. Now lspci's broader PCI-bus view wins, so on any host where PCI visibility is wider than device-cgroup visibility (shared bare-metal, some container/VM setups), examine will report every physically-present GPU as usable even when this process can only touch one of them. Left an inline note on the new scenario, but the fix likely needs to happen in probe_gpus_lspci or how gpus[]/has_amd_gpu are meant to be interpreted on hosts like this, not just in the test.
Reading lspci's processing-accelerator class is what finally finds Instinct parts, but it also changed *who decides* which GPUs exist, and the PCI bus answers a different question from the kernel. In a container the host bus still carries every card the node has while KFD describes only the one passed through, so `examine --json` listed eight MI300X GPUs where one was usable. That is what examine-16 caught on four lanes -- "the kernel describes 1 GPU node(s) but the report lists 8 AMD GPU(s)" -- and the test was right: `rocm serve` would have failed on a GPU `examine` had just advertised, which is worse than the under-detection this replaced, because that at least failed honestly. So keep the reported list as the set the kernel exposes and use the PCI scan to enrich it. `kfd_gpu_nodes_in` reads the KFD topology per node and returns each node's own gfx target alongside the PCI address it reports; `apply_kernel_gpu_membership` then matches those against what lspci found, by address rather than by position. A matched node keeps lspci's marketing name and APU/discrete verdict and fills only an absent target. A node the PCI scan did not list is still reported -- it is usable, so dropping it would under-report exactly the way this whole change exists to fix -- just without the enriched name. Devices on the bus that the kernel does not expose leave `gpus[]` but not the report: they are named in a note, because "the bus has it and the kernel does not" is itself a diagnosis. Two deliberate non-changes. An empty topology is a no-op rather than "no GPUs", so a host whose `amdgpu` never bound keeps its PCI list and the driver probes explain why nothing is usable. And only `is_amd` entries are touched, since KFD says nothing about an NVIDIA card. KFD states an address as `domain` plus `location_id`, the kernel's `pci_dev_id()` -- `(bus << 8) | devfn`. The decode is checked against hardware, not against itself: an 8-GPU MI300X host reports location_id 4352, 12032, 17920, 23808, 35584, 43520, 49664 and 55808, which decode to exactly the eight addresses `lspci -D` prints. A zero location_id is refused rather than decoded to the host bridge. This also removes the gfx-target gap-fill added in the previous commit, which review flagged as wired in by a statement no test could exercise. That is confirmed: deleting the call leaves the whole workspace suite green and turns the entire apparatus into four "never used" warnings, because the two unit tests called the inner function directly. It is not worth a test, because reading targets per node supersedes it -- each card takes the target of its own node, so the uniformity guard the gap-fill needed to avoid stamping an APU's target onto a dGPU is no longer a guard but a consequence. Covered by nine unit tests, including the eight-on-the-bus/one-in-the-kernel case and a planted-topology test that drives the sysfs read and the reconcile together. Verified end to end on an 8-GPU MI300X host: natively it reports all eight with their own addresses and targets and examine-16 passes, and with the topology bind-mounted down to a single node inside a mount namespace -- the container shape -- examine-16 fails on the previous commit with the exact CI message and passes here, reporting one GPU at 0000:5d:00.0 with the other seven noted. EAI-8449 Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Fixed in
Verified on hardware rather than reasoned about: the devbox is an 8-GPU MI300X host, so natively all 8 enumerate with their own addresses and Not verified, and worth saying plainly: the real CI container (its KFD filtering may differ from my bind-mount), MI350P, and any APU+dGPU host. I also removed the gfx-target gap-fill wiring the automated review flagged. Deleting it left the suite green and produced four Agreed on Strix Halo Ubuntu being the pre-existing VRAM-preflight issue; it is still failing across the queue for the same reason. |
Superseded: this objection was filed against an earlier revision. The branch has been re-reviewed at its current head and a fresh change request has been filed there, so this one is withdrawn to leave a single live objection to answer.
siloteemu
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 407d8fc
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Summary
Makes examine's AMD GPU list the set the KFD topology exposes, enriched (not decided) by the lspci scan matched on PCI address, after widening the lspci class filter to include processing accelerators. Outcome: Needs work — one reachable naming regression on the very container shape the change targets. Reviewed at check-run conclusions 20 success, 1 failure, 5 pending, 1 skipped. Verified: ran the ten new/affected unit tests in a throwaway copy (all pass) and mutated eleven individual branches of the production change one at a time in that copy — ten were killed by a named test, including three tests that die when address matching is reverted to positional, so the heart of this round is genuinely covered; the full suite, the end-to-end suite and any GPU or Windows lane were not run here. Also confirmed: the gap-fill apparatus from the middle commit is gone with no orphaned function, test or doc reference left behind; the new scenario id matches the feature file's convention and its naming lint; the expectations file has no row for it and its one adjacent exemption is unrelated and unwidened; no internal names, hosts or links in the diff; no injected instructions in code or commit messages. Blocking: 1 · Non-blocking: 5.
🚫 Blocking (must fix before merge)
crates/rocm-core/src/examine.rs:1341-1356 (paired with :1219-1221) — on a host with no lspci but with rocminfo, every GPU now ends up named "AMD GPU (from kernel topology)" and the report carries a note saying its marketing name is unknown, even though rocminfo knows the real name. The new membership pass runs before the rocminfo probe and pushes a placeholder-named entry for each kernel node the PCI scan could not name; the rocminfo probe only adopts a marketing name when the existing entry's name is empty ("if !marketing.is_empty() && gpu.name.is_empty()"), so the placeholder wins. Before this change the list was still empty at that point and rocminfo supplied the real names. That configuration is not exotic — it is the ordinary ROCm container, which ships rocminfo but frequently not pciutils, i.e. exactly the shape this PR exists to fix; the head commit's own text cites a host where lspci is absent. The "marketing name is unknown" note is also emitted before rocminfo has had a chance to speak, so it is asserted prematurely regardless. It blocks because it is a user-visible quality regression introduced by this diff on the targeted scenario, and no test covers the ordering interaction between the two probes. Concrete fix: let rocminfo replace a kernel-topology placeholder as well as an empty name (e.g. treat the placeholder as "unnamed" in that condition, or leave the name empty in the membership pass and render a default at report time), and emit the unnamed note only once naming is final. Add a unit test that runs the membership pass with an empty PCI list and then applies a rocminfo reading, asserting the marketing name survives and no "unknown name" note is produced.
Non-blocking
crates/rocm-core/src/examine.rs:1400-1403— the documented rule that an empty PCI address "matches nothing, not a wildcard" is the one branch that survived mutation: deleting the emptiness check leaves all ten tests green, so the guard is unproven (today unreachable from the lspci path, which always yields an address, but the doc states it as a rule).crates/rocm-core/src/diagnose.rs:1127-1131— the code comment says several KFD nodes may legitimately share one card's address on a partitioned Instinct; each such node clones the same PCI entry, so the AMD GPU count becomes the partition count and can trip the multi-GPU IOMMU check on a single physical card.tests/e2e-cucumber/features/examine.feature:190— the new scenario is tagged only for GPU, so it also runs on the Windows GPU lane and passes there only because a Linux-only sysfs read happens to fail; the repo's own capability code gates the equivalent read with an explicit platform cfg.crates/rocm-core/src/lib.rs:5145-5155— the node properties file is read twice per node (once for the target version, once for the address), and no test covers a node whose target comes from the standalone file while properties is absent, where the address silently becomes empty.tests/e2e-cucumber/features/examine.feature— the contributor rules ask for user-facing docs to move with observable behavior;examine --jsongains two new note forms and a changedgpus[]membership rule, and nothing in the README or the docs directory describes either (nothing is stale, but nothing documents the new contract either).
…cile
Review found the gfx-target work wired in by a statement no test could
exercise. Re-verified on this head: deleting `apply_kernel_gpu_membership(e,
&nodes)` left the whole workspace suite green -- 31 "test result: ok", zero
failures -- because every unit test called that inner function directly with a
hand-supplied node list, and examine-16 read only the entry count and PCI
address. Both halves of the fill were therefore unfalsifiable.
Two changes, one per half.
`probe_gpus_kernel_membership_in` gives the probe the same planted-directory
seam `kfd_gpu_nodes_in` already has, one level further out, so a test can drive
the read, the "cannot say" early return and the handoff together instead of only
the reconcile the handoff calls. The `/sys` path moves up into the probe, which
orphans `linux_kfd_gpu_nodes` -- its cfg split is now the probe's -- so it goes.
examine-16 now also asserts every AMD entry carries a non-empty `gfx_target`,
which is the observable end of the second half and what the scenario was missing
per AGENTS.md ("a unit test asserting the internal helper does NOT discharge
this; it proves the function, not the behavior"). Gated on the topology being
uniform, on the same premise-failure footing as the existing guards: where nodes
disagree, a node count cannot say which target belongs to which card, so the
step stays silent rather than assert something it has not established.
Falsified both. Deleting the handoff fails
`the_membership_probe_reconciles_the_topology_it_is_pointed_at` (left: 8, right:
1). Deleting the gap-fill on an 8-GPU gfx942 host with no `rocminfo` leaves all
eight entries with an empty `gfx_target`: the pre-existing count/PCI assertions
still pass, the new one fails.
EAI-8449
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…sing-accelerators Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…ses them The Windows build failed with three dead-code errors. probe_gpus_kernel_membership is cfg(any(target_os = "linux", test)), so off Linux it is the no-op stub and KfdGpuNode, apply_kernel_gpu_membership and pci_ids_match have no reachable caller -- but they carried no cfg of their own, so they were compiled and immediately unused. Linux clippy cannot see this: there the real branch compiles and every one of them is live. Gates them with the same cfg their only callers carry, and makes the stub a const fn, which clippy asks for once it is the definition that compiles. Verified by flipping the feature's cfgs from linux to windows and running clippy on Linux: three "never constructed/used" errors before, none after. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
🔴 Automated review · pr-review-watcher · 5127ae1 This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer. SummaryMakes 🚫 Blocking (must fix before merge)1. 2. Non-blocking
|
Superseded by a new round at 5127ae1. The naming objection is restated there because it is still live at this head, together with a second finding, so there is only one change request to answer.
siloteemu
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 5127ae1
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Two blocking findings at this head. The full round, including five non-blocking points, is in the comment on this PR.
1. crates/rocm-core/src/examine.rs:1219 (with :1371-1380 and :1413-1418) — the naming regression raised last round is still live. Nothing in this round touched the interaction. The membership pass still runs before probe_gpus_rocminfo (ordering at :395-404) and still pushes name: "AMD GPU (from kernel topology)" for every kernel node the PCI scan could not name. probe_gpus_rocminfo adopts a marketing name only when the existing name is empty — if !marketing.is_empty() && gpu.name.is_empty() — so the placeholder wins and the real name from rocminfo is discarded. The "their marketing name is unknown" note at :1413-1418 is likewise emitted before rocminfo has spoken, so it is asserted prematurely in every case.
The affected shape is the ordinary ROCm container: rocminfo present, pciutils absent — the same shape this PR's own scope caveat describes. Before this change the list was empty at that point and rocminfo supplied real names, so this is a user-visible regression the diff introduces on its own target scenario. It is unpinned in either direction: widening :1219 to replace the placeholder as well as an empty name leaves all 423 tests green, so no test would notice the bug or a fix.
Fix: treat the kernel-topology placeholder as unnamed in that condition (or leave the name empty in the membership pass and render a default at report time), and emit the unnamed note only after naming is final. Add a unit test that runs the membership pass against an empty PCI list, then applies a rocminfo reading, and asserts the marketing name survives and no "unknown name" note is produced.
2. crates/rocm-core/src/examine.rs:3303 — the_membership_probe_leaves_the_report_alone_when_the_topology_is_unreadable does not test the code it names. The test's comment names the early return in probe_gpus_kernel_membership_in — "No KFD at all … is 'cannot say', not 'no GPUs'" — but replacing that early return (:1326-1328) with unwrap_or_default(), which turns "cannot read" into "read an empty topology", leaves all 423 tests green. The assertion is satisfied entirely by the separate if nodes.is_empty() { return; } guard at the top of apply_kernel_gpu_membership, a pre-existing branch the test does not name. So the distinction the test exists to protect — unreadable versus empty — is unprotected, and a future refactor collapsing the two would pass.
Fix: make the assertion discriminate — drive apply_kernel_gpu_membership with an empty node list and the probe with an unreadable directory in the same test and assert both are no-ops for their own reasons, or have the unreadable case assert on something the empty-guard path cannot produce (the absence of the note text, plus an observable marker that the reconcile was never entered).
A test that does not test the thing it names gates a merge here rather than being filed as a nit, because a weak test is not strengthened after it lands.
The positive half: the class widening, the membership handoff and the gfx-target gap-fill are each killed by a named test under single-line mutation, the cfg gating added this round is correct, the removed topology helper leaves no orphaned reference, and the library is clean under clippy -D warnings.
…sing-accelerators Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…r (EAI-8449) The membership pass runs before probe_gpus_rocminfo and stamped "AMD GPU (from kernel topology)" on every kernel node the PCI scan could not name. rocminfo then adopted a marketing name only when the existing one was empty, so the placeholder won and the real model name was discarded -- on the ordinary ROCm container with rocminfo but no pciutils, which is the shape this PR exists for. Before the membership pass that list was empty there and rocminfo supplied real names, so the diff regressed its own target case. The placeholder is now one named constant that gpu_name_is_unknown() reads back, so a later probe treats it as the absence of a name that it is. The "marketing name unknown" note moved after rocminfo, since it was previously asserted before the probe most likely to refute it had spoken. the_membership_probe_leaves_the_report_alone_when_the_topology_is_unreadable named an early return it did not test: replacing that return with unwrap_or_default(), collapsing "cannot say" into "KFD listed no GPU", left all 423 tests green because apply_kernel_gpu_membership's own empty-guard absorbed it. probe_gpus_kernel_membership_in now reports which answer it got, so the test can discriminate. Falsified: reverting the name condition fails the new placeholder test; the unwrap_or_default() collapse fails the unreadable test. Each mutation reddens only its own test. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Numbering collision with #444, flagged here because neither PR will show a conflict. Both this PR and #444 add a scenario numbered #444 is approved and this one is not, so on current state #444 lands first and this PR wants Raising it rather than changing it, since the branch is yours. Happy to push the renumber if you would rather not carry it. |
juhovainio
left a comment
There was a problem hiding this comment.
The bug from my earlier review is fixed, and fixed well. That review flagged: on a shared bare-metal host (an 8-GPU MI300X node passing through only one GPU to a container), lspci walks the whole PCI bus and sees every physical card, while KFD/render-node access is scoped to what's actually usable — so examine would report GPUs the process can't actually touch.
Commit 407d8fc restructures the fix correctly: instead of PCI deciding which GPUs exist and KFD only filling gaps, the kernel topology now decides membership (which respects cgroup/passthrough restrictions), with PCI just supplying the marketing name for matched devices by PCI address — not position, which also fixes a latent bug on partitioned Instinct hosts where several KFD nodes can share one physical card's address. Unexposed-but-on-bus devices are kept in a note rather than silently dropped, which is the right call ("the bus has it and the kernel doesn't" is itself a diagnosis). The two follow-up commits (Windows dead-code cfg gating, rocminfo-name-beats-placeholder precedence) are both clean, narrow fixes.
Verified: read the reconciliation logic and its tests in full, ran cargo test -p rocm-core --lib examine at this head myself (53 passed, 0 failed). Real-hardware CI confirms it in practice — the MI300X and rad3 R9700 e2e runs that failed under the old logic ("kernel describes 1 GPU node but the report lists 8 AMD GPU(s)") are now green. Approving.
|
🔴 Automated review · pr-review-watcher · 5d7c3cd This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer. SummaryWidens the lspci class filter so Instinct parts (PCI class 1200, "Processing accelerators") are enumerated, then makes the reported AMD GPU list the set the kernel's KFD topology exposes — named and enriched from the PCI scan, matched by address — and adds a pass that reports which kernel-sourced cards ended up unnamed. Outcome: Needs work — one ordering invariant that exists only in prose, introduced by the very commit that fixes the prior round's naming regression. Verified: ran the crate's library tests on a scratch copy (424 pass) and mutated fifteen individual branches of the change one at a time in that scratch copy — all mutation experiments ran on a scratch copy; the tree under review was never modified. Both blocking findings from earlier rounds are genuinely discharged: reverting the placeholder-versus-marketing-name condition is killed by a named new test, and replacing the topology read's early return with a lossy default is now killed by the test that names it, which the added result enum makes discriminating. Also killed by named tests: the class widening, address-based rather than positional matching, the never-overwrite rule on an already-resolved target, the empty-topology guard, the zero-address refusal, the unexposed-device note, and the node sort. Confirmed independently: the PCI address decode matches the three real addresses it claims; the base branch's scenario numbering stops one below the number added here and the repo's own naming lint requires sequential per-file numbering with suite-wide unique indexes; no internal names, hosts, links or registry paths anywhere in the diff; no injected instructions in code, comments or commit messages. Not checked in this round: the end-to-end suite, any GPU or Windows lane, and cross-compilation for a non-Linux target. Reviewed against 26 successful checks, 1 skipped and 1 pending, with zero failures. Blocking: 1 · Non-blocking: 5. 🚫 Blocking (must fix before merge)
The head commit correctly splits the "marketing name is unknown" note out of the membership pass into
The naming half of the same fix is properly pinned: narrowing Concrete fix: give the ordering an assertion rather than a comment. The cheapest form is a unit test that drives the three passes in the wrong order and asserts the bad outcome is impossible — but that only restates the prose. Better: make the invariant structural so it cannot be violated silently, e.g. have Non-blocking
On the other reviewer's numbering point: the rule as stated matches the repository's own naming lint, which requires per-file sequential numbering and suite-wide unique indexes, and this branch is correct in isolation — the base branch stops one number below and the new scenario is appended last. The interaction with the other pull request was not verified in this round. |
siloteemu
left a comment
There was a problem hiding this comment.
pr-review-watcher · 5d7c3cd
Requesting changes on one item. The rest of this round's findings are non-blocking and are in the round comment posted alongside this.
The ordering contract that makes the head commit's fix work is stated only in a doc comment, and every way of breaking it leaves the whole suite green.
The head commit is right to split the "marketing name is unknown" note out of the kernel-membership pass: that pass runs before the rocminfo probe, so it cannot know whether a card is finally unnamed. The new function's own doc states the constraint — it must run after the rocminfo probe and before the sysfs fallback, because the first resolves the placeholder and the second plants the same placeholder again with a note of its own. Both halves of that sentence are unenforced. Measured on a scratch copy, with the tree under review left unmodified:
- Moving the call back above the rocminfo probe — which reinstates exactly the premature-note defect this commit exists to remove — leaves all 424 library tests passing.
- Moving it below the sysfs fallback, the double-count the comment warns about, leaves all 424 passing.
- Deleting the call outright leaves all 424 passing.
The naming half of the same fix is properly pinned: narrowing the unknown-name predicate back to an emptiness check is killed by the new test that names it. But that test invokes the membership pass, the rocminfo fold and the note function by hand in the right order, so it proves the three functions and says nothing about the sequence the product actually runs.
Why this gates rather than waits: it is the shape the fix itself created. Before this commit the note was emitted inside a pass that tests drive directly; now it is a free-floating call whose correctness is entirely positional, with the position asserted in prose only. A user-visible note on the machine-readable report can therefore regress silently to firing on hosts where every card is named, and a weak test is not strengthened after a merge — nobody reopens a landed change to do it.
Suggested fix, with the caveat that matters most: the naive fix does not close this. Adding a unit test that calls the three functions in the documented order is what the existing test already does, and it is precisely what survived all three mutations — an assertion that assumes the order cannot observe it. Make the invariant structural instead, so it cannot be violated silently: have the note function require a value only the post-rocminfo stage can produce (the topology value is already threaded through and currently discarded), or fold the probe and the note into one small stage function so the sequence is a single unit. Either makes the three mutations above fail to compile or fail a test, rather than passing quietly.
Two smaller things in the same area are non-blocking but worth reading together with this, since they are the same pattern at lower stakes: deleting the kernel-membership call from the probe chain also leaves all 424 green, and the documented rule that an empty PCI address matches nothing rather than acting as a wildcard is unproven for the third round running.
…lback (EAI-8449) The position `note_unnamed_kernel_topology_gpus` must hold -- after `probe_gpus_rocminfo`, before `probe_gpus_sysfs_fallback` -- was stated only in a doc comment. Moving the call above `rocminfo` (which reinstates the premature-note defect the pass was split out to remove), moving it below the fallback, and deleting it outright each left all 424 tests passing: the existing coverage calls the passes directly in an order it chooses itself, so it cannot see the real call site running them in another. Gather the passes that follow the PCI scan into `probe_gpus_after_lspci`, so the order exists in exactly one place, and give it a `GpuProbeSources` seam for the three things it reads -- the KFD nodes directory, a `rocminfo` listing, and the sysfs gfx target -- so the real sequence is drivable against a planted host. All three mutations now fail. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
🔴 Automated review · pr-review-watcher · 3df0c04 This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer. SummaryMakes Prior round
🚫 Blocking (must fix before merge)None. Non-blocking
|
|
The collision is real and the rule you state is the right one — but the mechanism is not, and the difference matters for how it gets caught. Confirmed real. Not silent, though. Both scenarios are appended at the end of the same file, after The merged blob carries conflict markers at Worth correcting because "neither PR will show a conflict" invites merging both and trusting green, and the actual failure is earlier and louder than that. Why it cannot be pre-resolved. Renumbering one of them to Both are mine and both are approved as of this morning, so I will carry the renumber on whichever goes second — no need for you to push it. Thanks for catching it; a duplicate index is the kind of thing that only surfaces as a confusing |
…sing-accelerators Signed-off-by: Roman Sirokov <roman.sirokov@amd.com> # Conflicts: # tests/e2e-cucumber/features/examine.feature
|
Carried the renumber, as promised above —
As predicted in the thread above, git did not merge this silently: both scenarios append after Verified on the Linux devbox at the merge commit, not inferred:
Falsified rather than assumed: putting the index back to The merge also pulls in |
… first
HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.
The search stopped at the first match and looked for one library only, so
the second copy could not be seen at all. It is now one walker serving
both searches, recording every copy found, plus the same for the HIP
runtime so the two can be compared. Attributing each copy to the
installation that owns it is deferred; comgr_matches_runtime stays
permanently unset until that lands.
The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why the loader path the CLI sets for its own
processes walks that directory. Searching the root alone therefore missed
the copy this CLI installs itself, which is the case the entry exists for.
That knowledge now lives in one place. collect_managed_runtime_library_paths
is shared by the comgr search and the loader path, so a managed runtime's
layout is described once rather than twice. A second description is a
second thing to keep correct, and the two had already diverged.
Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.
The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record the predictable venv layout is read
directly, and a miss costs a directory that is simply not reported.
No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.
Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.
The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.
Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.
The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.
Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.
Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.
Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.
… first
HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.
The search stopped at the first match and looked for one library only, so
the second copy could not be seen at all. It is now one walker serving
both searches, recording every copy found, plus the same for the HIP
runtime so the two can be compared. Attributing each copy to the
installation that owns it is deferred; comgr_matches_runtime stays
permanently unset until that lands.
The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why the loader path the CLI sets for its own
processes walks that directory. Searching the root alone therefore missed
the copy this CLI installs itself, which is the case the entry exists for.
That knowledge now lives in one place. collect_managed_runtime_library_paths
is shared by the comgr search and the loader path, so a managed runtime's
layout is described once rather than twice. A second description is a
second thing to keep correct, and the two had already diverged.
Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.
The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record the predictable venv layout is read
directly, and a miss costs a directory that is simply not reported.
No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.
Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.
The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.
Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.
The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.
Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.
Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.
Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.
… first
HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.
The search stopped at the first match, so the second copy could not be
seen at all. It now records every copy found, in loader search order.
Attributing each copy to the installation that owns it, and doing the
same search for the HIP runtime so the two can be compared, is deferred;
comgr_matches_runtime stays permanently unset until that lands.
The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why `managed_sdk_ld_library_path` (the
environment the CLI builds to run an SDK tool, such as
rocm_agent_enumerator, inside a managed runtime) walks that directory.
Searching the root alone therefore missed the copy this CLI installs
itself, which is the case the entry exists for.
That knowledge now lives in one place for this search:
collect_managed_runtime_library_paths is shared by the comgr search here
and by managed_sdk_ld_library_path, so that one layout is described once
there rather than twice. A third description, collect_runtime_environment_paths,
still builds the environment a served engine process gets; unifying all
three is follow-up work, not part of this search.
Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.
The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record, root's own `_rocm_sdk_*` name is read
to find its parent -- the site-packages directory its siblings share --
and a miss costs a directory that is simply not reported.
No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.
Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.
The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.
Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.
The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.
Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.
Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.
Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.
Review: four more issues. (1) comgr_selected and the multi-copy note named
the ld.so.cache system copy as "would load" whenever a managed wheel
runtime was active but this process's own LD_LIBRARY_PATH was empty --
wrong, because rocm serve/rocm chat prepend the active runtime's own
library directories onto LD_LIBRARY_PATH before launching the engine, so
the wheel copy is what a served process actually loads. probe_comgr now
takes the active runtime's interpreter (the same FrameworkInterpreter
already threaded through probe_framework) and searches its library_paths
first, ahead of the ambient environment; pinned by
the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The
site_packages: None fallback in collect_managed_runtime_library_paths
guessed a venv layout (root/lib/<python>/site-packages) that the installer
never produces -- root is one of the runtime's own _rocm_sdk_* package
directories, not a venv root, so the guess always found nothing. It now
reads root.parent() instead, which is the site-packages directory the
guessed layout could never reach. The test fixture
(wheel_runtime_on_disk) was built from the same wrong venv shape and so
could not have caught this; rebuilt it from the real installer layout
(ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed
a_wheel_runtime_with_no_registry_record_is_still_found fails against the
old fallback before passing against the fix. (3) probe_comgr,
comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so
the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the
selection, and the multi-copy note could all break with the rest of the
suite still green. Split the ordering/selection logic into a pure
find_comgr_copies, the ldconfig -p line parsing into
parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings
(opt directory taken as a parameter rather than hardcoded, so a test can
point it at a temp folder instead of the real /opt on the machine running
the suite); each is unit-tested directly, and each new test was confirmed
to fail against a deliberately broken version of the function it covers
before being restored. (4) This message previously claimed a HIP copy
list and a single shared description of the loader path launched
processes use; neither matches this diff. Corrected above.
Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
… first
HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.
The search stopped at the first match, so the second copy could not be
seen at all. It now records every copy found, in loader search order.
Attributing each copy to the installation that owns it, and doing the
same search for the HIP runtime so the two can be compared, is deferred;
comgr_matches_runtime stays permanently unset until that lands.
The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why `managed_sdk_ld_library_path` (the
environment the CLI builds to run an SDK tool, such as
rocm_agent_enumerator, inside a managed runtime) walks that directory.
Searching the root alone therefore missed the copy this CLI installs
itself, which is the case the entry exists for.
That knowledge now lives in one place for this search:
collect_managed_runtime_library_paths is shared by the comgr search here
and by managed_sdk_ld_library_path, so that one layout is described once
there rather than twice. A third description, collect_runtime_environment_paths,
still builds the environment a served engine process gets; unifying all
three is follow-up work, not part of this search.
Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.
The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record, root's own `_rocm_sdk_*` name is read
to find its parent -- the site-packages directory its siblings share --
and a miss costs a directory that is simply not reported.
No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.
Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.
The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.
Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.
The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.
Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.
Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.
Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.
Review: four more issues. (1) comgr_selected and the multi-copy note named
the ld.so.cache system copy as "would load" whenever a managed wheel
runtime was active but this process's own LD_LIBRARY_PATH was empty --
wrong, because rocm serve/rocm chat prepend the active runtime's own
library directories onto LD_LIBRARY_PATH before launching the engine, so
the wheel copy is what a served process actually loads. probe_comgr now
takes the active runtime's interpreter (the same FrameworkInterpreter
already threaded through probe_framework) and searches its library_paths
first, ahead of the ambient environment; pinned by
the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The
site_packages: None fallback in collect_managed_runtime_library_paths
guessed a venv layout (root/lib/<python>/site-packages) that the installer
never produces -- root is one of the runtime's own _rocm_sdk_* package
directories, not a venv root, so the guess always found nothing. It now
reads root.parent() instead, which is the site-packages directory the
guessed layout could never reach. The test fixture
(wheel_runtime_on_disk) was built from the same wrong venv shape and so
could not have caught this; rebuilt it from the real installer layout
(ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed
a_wheel_runtime_with_no_registry_record_is_still_found fails against the
old fallback before passing against the fix. (3) probe_comgr,
comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so
the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the
selection, and the multi-copy note could all break with the rest of the
suite still green. Split the ordering/selection logic into a pure
find_comgr_copies, the ldconfig -p line parsing into
parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings
(opt directory taken as a parameter rather than hardcoded, so a test can
point it at a temp folder instead of the real /opt on the machine running
the suite); each is unit-tested directly, and each new test was confirmed
to fail against a deliberately broken version of the function it covers
before being restored. (4) This message previously claimed a HIP copy
list and a single shared description of the loader path launched
processes use; neither matches this diff. Corrected above.
Review: a second round, one automated and one human, reaching the same
place by different routes.
The human reviewer confirmed, independently, that the site_packages:
None fallback added in the prior round could not actually fire:
ROCM_SDK_PROBE_SCRIPT has recorded site_packages unconditionally, outside
its try, since the probe's initial version, so every manifest that
parses at all carries Some there. The test built to cover it could not
have caught that either way, since it called the function directly with
None rather than through a real candidate. Rather than keep a reachable-
looking branch and a comment describing behaviour the code does not
have, the None arm is now a no-op (root's own directories, already
collected unconditionally above, are what a hand-constructed None caller
still gets) and the test is removed. The neighbouring root-format test
is kept and re-targeted: it already called the same function directly
with None, so it is what actually exercises that arm's contract, not the
removed test.
The automated reviewer caught a real regression the first round
introduced: three new unit tests feed a path into the `:`-split
LD_LIBRARY_PATH parser without a unix cfg gate, same as the pattern the
file already uses for that reason, and that broke the required
windows-build-and-test lane. Gated, along with the helper they share
(which would otherwise be unused, and clippy-denied, once its only
callers are gated).
It also found that the active-runtime-priority fix regresses
examine-finds-the-managed-runtimes-own-compilation-library on a healthy
host: once a managed runtime is active, its directory is found first and
labelled active-runtime, and the dedup this search has always done on
resolved path then drops the later managed-runtime hit for the identical
file -- the scenario's step asserted on the label that fix makes
unreachable. The step now accepts either label as proof the CLI's own
installed copy was found, and a new unit test
(an_active_runtime_directory_that_is_also_a_managed_root_keeps_one_label)
pins that an overlapping directory keeps exactly one label rather than
being reported twice; confirmed to fail when the active-runtime search is
moved after the others, then restored.
Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
… first
HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.
The search stopped at the first match, so the second copy could not be
seen at all. It now records every copy found, in loader search order.
Attributing each copy to the installation that owns it, and doing the
same search for the HIP runtime so the two can be compared, is deferred;
comgr_matches_runtime stays permanently unset until that lands.
The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why `managed_sdk_ld_library_path` (the
environment the CLI builds to run an SDK tool, such as
rocm_agent_enumerator, inside a managed runtime) walks that directory.
Searching the root alone therefore missed the copy this CLI installs
itself, which is the case the entry exists for.
That knowledge now lives in one place for this search:
collect_managed_runtime_library_paths is shared by the comgr search here
and by managed_sdk_ld_library_path, so that one layout is described once
there rather than twice. A third description, collect_runtime_environment_paths,
still builds the environment a served engine process gets; unifying all
three is follow-up work, not part of this search.
Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.
The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record, root's own `_rocm_sdk_*` name is read
to find its parent -- the site-packages directory its siblings share --
and a miss costs a directory that is simply not reported.
No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.
Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.
The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.
Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.
The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.
Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.
Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.
Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.
Review: four more issues. (1) comgr_selected and the multi-copy note named
the ld.so.cache system copy as "would load" whenever a managed wheel
runtime was active but this process's own LD_LIBRARY_PATH was empty --
wrong, because rocm serve/rocm chat prepend the active runtime's own
library directories onto LD_LIBRARY_PATH before launching the engine, so
the wheel copy is what a served process actually loads. probe_comgr now
takes the active runtime's interpreter (the same FrameworkInterpreter
already threaded through probe_framework) and searches its library_paths
first, ahead of the ambient environment; pinned by
the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The
site_packages: None fallback in collect_managed_runtime_library_paths
guessed a venv layout (root/lib/<python>/site-packages) that the installer
never produces -- root is one of the runtime's own _rocm_sdk_* package
directories, not a venv root, so the guess always found nothing. It now
reads root.parent() instead, which is the site-packages directory the
guessed layout could never reach. The test fixture
(wheel_runtime_on_disk) was built from the same wrong venv shape and so
could not have caught this; rebuilt it from the real installer layout
(ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed
a_wheel_runtime_with_no_registry_record_is_still_found fails against the
old fallback before passing against the fix. (3) probe_comgr,
comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so
the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the
selection, and the multi-copy note could all break with the rest of the
suite still green. Split the ordering/selection logic into a pure
find_comgr_copies, the ldconfig -p line parsing into
parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings
(opt directory taken as a parameter rather than hardcoded, so a test can
point it at a temp folder instead of the real /opt on the machine running
the suite); each is unit-tested directly, and each new test was confirmed
to fail against a deliberately broken version of the function it covers
before being restored. (4) This message previously claimed a HIP copy
list and a single shared description of the loader path launched
processes use; neither matches this diff. Corrected above.
Review: a second round, one automated and one human, reaching the same
place by different routes.
The human reviewer confirmed, independently, that the site_packages:
None fallback added in the prior round could not actually fire:
ROCM_SDK_PROBE_SCRIPT has recorded site_packages unconditionally, outside
its try, since the probe's initial version, so every manifest that
parses at all carries Some there. The test built to cover it could not
have caught that either way, since it called the function directly with
None rather than through a real candidate. Rather than keep a reachable-
looking branch and a comment describing behaviour the code does not
have, the None arm is now a no-op (root's own directories, already
collected unconditionally above, are what a hand-constructed None caller
still gets) and the test is removed. The neighbouring root-format test
is kept and re-targeted: it already called the same function directly
with None, so it is what actually exercises that arm's contract, not the
removed test.
The automated reviewer caught a real regression the first round
introduced: three new unit tests feed a path into the `:`-split
LD_LIBRARY_PATH parser without a unix cfg gate, same as the pattern the
file already uses for that reason, and that broke the required
windows-build-and-test lane. Gated, along with the helper they share
(which would otherwise be unused, and clippy-denied, once its only
callers are gated).
It also found that the active-runtime-priority fix regresses
examine-finds-the-managed-runtimes-own-compilation-library on a healthy
host: once a managed runtime is active, its directory is found first and
labelled active-runtime, and the dedup this search has always done on
resolved path then drops the later managed-runtime hit for the identical
file -- the scenario's step asserted on the label that fix makes
unreachable. The step now accepts either label as proof the CLI's own
installed copy was found, and a new unit test
(an_active_runtime_directory_that_is_also_a_managed_root_keeps_one_label)
pins that an overlapping directory keeps exactly one label rather than
being reported twice; confirmed to fail when the active-runtime search is
moved after the others, then restored.
Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
… first
HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.
The search stopped at the first match, so the second copy could not be
seen at all. It now records every copy found, in loader search order.
Attributing each copy to the installation that owns it, and doing the
same search for the HIP runtime so the two can be compared, is deferred;
comgr_matches_runtime stays permanently unset until that lands.
The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why `managed_sdk_ld_library_path` (the
environment the CLI builds to run an SDK tool, such as
rocm_agent_enumerator, inside a managed runtime) walks that directory.
Searching the root alone therefore missed the copy this CLI installs
itself, which is the case the entry exists for.
That knowledge now lives in one place for this search:
collect_managed_runtime_library_paths is shared by the comgr search here
and by managed_sdk_ld_library_path, so that one layout is described once
there rather than twice. A third description, collect_runtime_environment_paths,
still builds the environment a served engine process gets; unifying all
three is follow-up work, not part of this search.
Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.
The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record, root's own `_rocm_sdk_*` name is read
to find its parent -- the site-packages directory its siblings share --
and a miss costs a directory that is simply not reported.
No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.
Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.
The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.
Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.
The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.
Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.
Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.
Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.
Review: four more issues. (1) comgr_selected and the multi-copy note named
the ld.so.cache system copy as "would load" whenever a managed wheel
runtime was active but this process's own LD_LIBRARY_PATH was empty --
wrong, because rocm serve/rocm chat prepend the active runtime's own
library directories onto LD_LIBRARY_PATH before launching the engine, so
the wheel copy is what a served process actually loads. probe_comgr now
takes the active runtime's interpreter (the same FrameworkInterpreter
already threaded through probe_framework) and searches its library_paths
first, ahead of the ambient environment; pinned by
the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The
site_packages: None fallback in collect_managed_runtime_library_paths
guessed a venv layout (root/lib/<python>/site-packages) that the installer
never produces -- root is one of the runtime's own _rocm_sdk_* package
directories, not a venv root, so the guess always found nothing. It now
reads root.parent() instead, which is the site-packages directory the
guessed layout could never reach. The test fixture
(wheel_runtime_on_disk) was built from the same wrong venv shape and so
could not have caught this; rebuilt it from the real installer layout
(ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed
a_wheel_runtime_with_no_registry_record_is_still_found fails against the
old fallback before passing against the fix. (3) probe_comgr,
comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so
the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the
selection, and the multi-copy note could all break with the rest of the
suite still green. Split the ordering/selection logic into a pure
find_comgr_copies, the ldconfig -p line parsing into
parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings
(opt directory taken as a parameter rather than hardcoded, so a test can
point it at a temp folder instead of the real /opt on the machine running
the suite); each is unit-tested directly, and each new test was confirmed
to fail against a deliberately broken version of the function it covers
before being restored. (4) This message previously claimed a HIP copy
list and a single shared description of the loader path launched
processes use; neither matches this diff. Corrected above.
Review: a second round, one automated and one human, reaching the same
place by different routes.
The human reviewer confirmed, independently, that the site_packages:
None fallback added in the prior round could not actually fire:
ROCM_SDK_PROBE_SCRIPT has recorded site_packages unconditionally, outside
its try, since the probe's initial version, so every manifest that
parses at all carries Some there. The test built to cover it could not
have caught that either way, since it called the function directly with
None rather than through a real candidate. Rather than keep a reachable-
looking branch and a comment describing behaviour the code does not
have, the None arm is now a no-op (root's own directories, already
collected unconditionally above, are what a hand-constructed None caller
still gets) and the test is removed. The neighbouring root-format test
is kept and re-targeted: it already called the same function directly
with None, so it is what actually exercises that arm's contract, not the
removed test.
The automated reviewer caught a real regression the first round
introduced: three new unit tests feed a path into the `:`-split
LD_LIBRARY_PATH parser without a unix cfg gate, same as the pattern the
file already uses for that reason, and that broke the required
windows-build-and-test lane. Gated, along with the helper they share
(which would otherwise be unused, and clippy-denied, once its only
callers are gated).
It also found that the active-runtime-priority fix regresses
examine-finds-the-managed-runtimes-own-compilation-library on a healthy
host: once a managed runtime is active, its directory is found first and
labelled active-runtime, and the dedup this search has always done on
resolved path then drops the later managed-runtime hit for the identical
file -- the scenario's step asserted on the label that fix makes
unreachable. The step now accepts either label as proof the CLI's own
installed copy was found, and a new unit test
(an_active_runtime_directory_that_is_also_a_managed_root_keeps_one_label)
pins that an overlapping directory keeps exactly one label rather than
being reported twice; confirmed to fail when the active-runtime search is
moved after the others, then restored.
Review: a third round, three issues.
(1) comgr_matches_runtime was declared, defaulted, and listed in the
expected-keys test, but nothing in this diff computes it -- comparing
comgr's copies against the HIP runtime's is still deferred, same as the
commit message already said. A reader of the JSON contract alone had no
way to know the field was permanently unset rather than merely not yet
exercised by a test. Dropped from this commit entirely (the struct field,
its Default entry, its expected-keys assertion); `serde(default)` makes
re-adding it free once something actually computes it.
(2) comgr_selected named the first entry in comgr_paths as "would load"
regardless of which tier found it, so a machine carrying only a leftover
/opt/rocm-* install or an inactive managed runtime -- never a copy on
active-runtime, ld-library-path, or loader-cache -- was told that stale
copy would load, when the loader does not search /opt or a managed
runtime's directory on its own. find_comgr_copies now selects only from
those three loader-consulted tiers (LOADER_PATH_SOURCES), leaving
comgr_selected None and rewording the note when every hit is
rocm-install/managed-runtime-only; pinned by
a_search_dir_only_hit_is_not_reported_as_the_one_that_would_load. The e2e
step (assert_comgr_selection_is_stated) now asserts the same tier rule
against the real JSON: selected absent plus only non-loader-tier copies
is the pass case, not a shape to tolerate.
(3) comgr_search_dirs and managed_runtime_roots had no unit tests of
their own, and swapping the loader-cache and search-dir loops inside
find_comgr_copies left every then-existing test green -- nothing pinned
their relative order. Split comgr_search_dirs into a pure
comgr_search_dirs_in (rocm_path, the /opt directory, and the resolved
managed runtimes all taken as parameters) and added
comgr_search_dirs_labels_each_tier, covering all three source labels
against temp directories. Added loader_cache_outranks_search_dirs, built
so an ld-library-path hit cannot mask the order under test. Confirmed the
claim directly: swapped the two loops, reran the suite -- exactly
loader_cache_outranks_search_dirs failed, the other 65 tests (including
ld_library_path_outranks_loader_cache_and_search_dirs) stayed green --
then reverted the swap and reran clean before committing.
Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
… first
HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.
The search stopped at the first match, so the second copy could not be
seen at all. It now records every copy found, in loader search order.
Attributing each copy to the installation that owns it, and doing the
same search for the HIP runtime so the two can be compared, is deferred;
comgr_matches_runtime stays permanently unset until that lands.
The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why `managed_sdk_ld_library_path` (the
environment the CLI builds to run an SDK tool, such as
rocm_agent_enumerator, inside a managed runtime) walks that directory.
Searching the root alone therefore missed the copy this CLI installs
itself, which is the case the entry exists for.
That knowledge now lives in one place for this search:
collect_managed_runtime_library_paths is shared by the comgr search here
and by managed_sdk_ld_library_path, so that one layout is described once
there rather than twice. A third description, collect_runtime_environment_paths,
still builds the environment a served engine process gets; unifying all
three is follow-up work, not part of this search.
Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.
The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record, root's own `_rocm_sdk_*` name is read
to find its parent -- the site-packages directory its siblings share --
and a miss costs a directory that is simply not reported.
No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.
Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.
The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.
Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.
The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.
Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.
Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.
Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.
Review: four more issues. (1) comgr_selected and the multi-copy note named
the ld.so.cache system copy as "would load" whenever a managed wheel
runtime was active but this process's own LD_LIBRARY_PATH was empty --
wrong, because rocm serve/rocm chat prepend the active runtime's own
library directories onto LD_LIBRARY_PATH before launching the engine, so
the wheel copy is what a served process actually loads. probe_comgr now
takes the active runtime's interpreter (the same FrameworkInterpreter
already threaded through probe_framework) and searches its library_paths
first, ahead of the ambient environment; pinned by
the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The
site_packages: None fallback in collect_managed_runtime_library_paths
guessed a venv layout (root/lib/<python>/site-packages) that the installer
never produces -- root is one of the runtime's own _rocm_sdk_* package
directories, not a venv root, so the guess always found nothing. It now
reads root.parent() instead, which is the site-packages directory the
guessed layout could never reach. The test fixture
(wheel_runtime_on_disk) was built from the same wrong venv shape and so
could not have caught this; rebuilt it from the real installer layout
(ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed
a_wheel_runtime_with_no_registry_record_is_still_found fails against the
old fallback before passing against the fix. (3) probe_comgr,
comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so
the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the
selection, and the multi-copy note could all break with the rest of the
suite still green. Split the ordering/selection logic into a pure
find_comgr_copies, the ldconfig -p line parsing into
parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings
(opt directory taken as a parameter rather than hardcoded, so a test can
point it at a temp folder instead of the real /opt on the machine running
the suite); each is unit-tested directly, and each new test was confirmed
to fail against a deliberately broken version of the function it covers
before being restored. (4) This message previously claimed a HIP copy
list and a single shared description of the loader path launched
processes use; neither matches this diff. Corrected above.
Review: a second round, one automated and one human, reaching the same
place by different routes.
The human reviewer confirmed, independently, that the site_packages:
None fallback added in the prior round could not actually fire:
ROCM_SDK_PROBE_SCRIPT has recorded site_packages unconditionally, outside
its try, since the probe's initial version, so every manifest that
parses at all carries Some there. The test built to cover it could not
have caught that either way, since it called the function directly with
None rather than through a real candidate. Rather than keep a reachable-
looking branch and a comment describing behaviour the code does not
have, the None arm is now a no-op (root's own directories, already
collected unconditionally above, are what a hand-constructed None caller
still gets) and the test is removed. The neighbouring root-format test
is kept and re-targeted: it already called the same function directly
with None, so it is what actually exercises that arm's contract, not the
removed test.
The automated reviewer caught a real regression the first round
introduced: three new unit tests feed a path into the `:`-split
LD_LIBRARY_PATH parser without a unix cfg gate, same as the pattern the
file already uses for that reason, and that broke the required
windows-build-and-test lane. Gated, along with the helper they share
(which would otherwise be unused, and clippy-denied, once its only
callers are gated).
It also found that the active-runtime-priority fix regresses
examine-finds-the-managed-runtimes-own-compilation-library on a healthy
host: once a managed runtime is active, its directory is found first and
labelled active-runtime, and the dedup this search has always done on
resolved path then drops the later managed-runtime hit for the identical
file -- the scenario's step asserted on the label that fix makes
unreachable. The step now accepts either label as proof the CLI's own
installed copy was found, and a new unit test
(an_active_runtime_directory_that_is_also_a_managed_root_keeps_one_label)
pins that an overlapping directory keeps exactly one label rather than
being reported twice; confirmed to fail when the active-runtime search is
moved after the others, then restored.
Review: a third round, three issues.
(1) comgr_matches_runtime was declared, defaulted, and listed in the
expected-keys test, but nothing in this diff computes it -- comparing
comgr's copies against the HIP runtime's is still deferred, same as the
commit message already said. A reader of the JSON contract alone had no
way to know the field was permanently unset rather than merely not yet
exercised by a test. Dropped from this commit entirely (the struct field,
its Default entry, its expected-keys assertion); `serde(default)` makes
re-adding it free once something actually computes it.
(2) comgr_selected named the first entry in comgr_paths as "would load"
regardless of which tier found it, so a machine carrying only a leftover
/opt/rocm-* install or an inactive managed runtime -- never a copy on
active-runtime, ld-library-path, or loader-cache -- was told that stale
copy would load, when the loader does not search /opt or a managed
runtime's directory on its own. find_comgr_copies now selects only from
those three loader-consulted tiers (LOADER_PATH_SOURCES), leaving
comgr_selected None and rewording the note when every hit is
rocm-install/managed-runtime-only; pinned by
a_search_dir_only_hit_is_not_reported_as_the_one_that_would_load. The e2e
step (assert_comgr_selection_is_stated) now asserts the same tier rule
against the real JSON: selected absent plus only non-loader-tier copies
is the pass case, not a shape to tolerate.
(3) comgr_search_dirs and managed_runtime_roots had no unit tests of
their own, and swapping the loader-cache and search-dir loops inside
find_comgr_copies left every then-existing test green -- nothing pinned
their relative order. Split comgr_search_dirs into a pure
comgr_search_dirs_in (rocm_path, the /opt directory, and the resolved
managed runtimes all taken as parameters) and added
comgr_search_dirs_labels_each_tier, covering all three source labels
against temp directories. Added loader_cache_outranks_search_dirs, built
so an ld-library-path hit cannot mask the order under test. Confirmed the
claim directly: swapped the two loops, reran the suite -- exactly
loader_cache_outranks_search_dirs failed, the other 65 tests (including
ld_library_path_outranks_loader_cache_and_search_dirs) stayed green --
then reverted the swap and reran clean before committing.
Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
… first
HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.
The search stopped at the first match, so the second copy could not be
seen at all. It now records every copy found, in loader search order.
Attributing each copy to the installation that owns it, and doing the
same search for the HIP runtime so the two can be compared, is deferred;
comgr_matches_runtime stays permanently unset until that lands.
The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why `managed_sdk_ld_library_path` (the
environment the CLI builds to run an SDK tool, such as
rocm_agent_enumerator, inside a managed runtime) walks that directory.
Searching the root alone therefore missed the copy this CLI installs
itself, which is the case the entry exists for.
That knowledge now lives in one place for this search:
collect_managed_runtime_library_paths is shared by the comgr search here
and by managed_sdk_ld_library_path, so that one layout is described once
there rather than twice. A third description, collect_runtime_environment_paths,
still builds the environment a served engine process gets; unifying all
three is follow-up work, not part of this search.
Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.
The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record, root's own `_rocm_sdk_*` name is read
to find its parent -- the site-packages directory its siblings share --
and a miss costs a directory that is simply not reported.
No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.
Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.
The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.
Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.
The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.
Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.
Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.
Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.
Review: four more issues. (1) comgr_selected and the multi-copy note named
the ld.so.cache system copy as "would load" whenever a managed wheel
runtime was active but this process's own LD_LIBRARY_PATH was empty --
wrong, because rocm serve/rocm chat prepend the active runtime's own
library directories onto LD_LIBRARY_PATH before launching the engine, so
the wheel copy is what a served process actually loads. probe_comgr now
takes the active runtime's interpreter (the same FrameworkInterpreter
already threaded through probe_framework) and searches its library_paths
first, ahead of the ambient environment; pinned by
the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The
site_packages: None fallback in collect_managed_runtime_library_paths
guessed a venv layout (root/lib/<python>/site-packages) that the installer
never produces -- root is one of the runtime's own _rocm_sdk_* package
directories, not a venv root, so the guess always found nothing. It now
reads root.parent() instead, which is the site-packages directory the
guessed layout could never reach. The test fixture
(wheel_runtime_on_disk) was built from the same wrong venv shape and so
could not have caught this; rebuilt it from the real installer layout
(ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed
a_wheel_runtime_with_no_registry_record_is_still_found fails against the
old fallback before passing against the fix. (3) probe_comgr,
comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so
the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the
selection, and the multi-copy note could all break with the rest of the
suite still green. Split the ordering/selection logic into a pure
find_comgr_copies, the ldconfig -p line parsing into
parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings
(opt directory taken as a parameter rather than hardcoded, so a test can
point it at a temp folder instead of the real /opt on the machine running
the suite); each is unit-tested directly, and each new test was confirmed
to fail against a deliberately broken version of the function it covers
before being restored. (4) This message previously claimed a HIP copy
list and a single shared description of the loader path launched
processes use; neither matches this diff. Corrected above.
Review: a second round, one automated and one human, reaching the same
place by different routes.
The human reviewer confirmed, independently, that the site_packages:
None fallback added in the prior round could not actually fire:
ROCM_SDK_PROBE_SCRIPT has recorded site_packages unconditionally, outside
its try, since the probe's initial version, so every manifest that
parses at all carries Some there. The test built to cover it could not
have caught that either way, since it called the function directly with
None rather than through a real candidate. Rather than keep a reachable-
looking branch and a comment describing behaviour the code does not
have, the None arm is now a no-op (root's own directories, already
collected unconditionally above, are what a hand-constructed None caller
still gets) and the test is removed. The neighbouring root-format test
is kept and re-targeted: it already called the same function directly
with None, so it is what actually exercises that arm's contract, not the
removed test.
The automated reviewer caught a real regression the first round
introduced: three new unit tests feed a path into the `:`-split
LD_LIBRARY_PATH parser without a unix cfg gate, same as the pattern the
file already uses for that reason, and that broke the required
windows-build-and-test lane. Gated, along with the helper they share
(which would otherwise be unused, and clippy-denied, once its only
callers are gated).
It also found that the active-runtime-priority fix regresses
examine-finds-the-managed-runtimes-own-compilation-library on a healthy
host: once a managed runtime is active, its directory is found first and
labelled active-runtime, and the dedup this search has always done on
resolved path then drops the later managed-runtime hit for the identical
file -- the scenario's step asserted on the label that fix makes
unreachable. The step now accepts either label as proof the CLI's own
installed copy was found, and a new unit test
(an_active_runtime_directory_that_is_also_a_managed_root_keeps_one_label)
pins that an overlapping directory keeps exactly one label rather than
being reported twice; confirmed to fail when the active-runtime search is
moved after the others, then restored.
Review: a third round, three issues.
(1) comgr_matches_runtime was declared, defaulted, and listed in the
expected-keys test, but nothing in this diff computes it -- comparing
comgr's copies against the HIP runtime's is still deferred, same as the
commit message already said. A reader of the JSON contract alone had no
way to know the field was permanently unset rather than merely not yet
exercised by a test. Dropped from this commit entirely (the struct field,
its Default entry, its expected-keys assertion); `serde(default)` makes
re-adding it free once something actually computes it.
(2) comgr_selected named the first entry in comgr_paths as "would load"
regardless of which tier found it, so a machine carrying only a leftover
/opt/rocm-* install or an inactive managed runtime -- never a copy on
active-runtime, ld-library-path, or loader-cache -- was told that stale
copy would load, when the loader does not search /opt or a managed
runtime's directory on its own. find_comgr_copies now selects only from
those three loader-consulted tiers (LOADER_PATH_SOURCES), leaving
comgr_selected None and rewording the note when every hit is
rocm-install/managed-runtime-only; pinned by
a_search_dir_only_hit_is_not_reported_as_the_one_that_would_load. The e2e
step (assert_comgr_selection_is_stated) now asserts the same tier rule
against the real JSON: selected absent plus only non-loader-tier copies
is the pass case, not a shape to tolerate.
(3) comgr_search_dirs and managed_runtime_roots had no unit tests of
their own, and swapping the loader-cache and search-dir loops inside
find_comgr_copies left every then-existing test green -- nothing pinned
their relative order. Split comgr_search_dirs into a pure
comgr_search_dirs_in (rocm_path, the /opt directory, and the resolved
managed runtimes all taken as parameters) and added
comgr_search_dirs_labels_each_tier, covering all three source labels
against temp directories. Added loader_cache_outranks_search_dirs, built
so an ld-library-path hit cannot mask the order under test. Confirmed the
claim directly: swapped the two loops, reran the suite -- exactly
loader_cache_outranks_search_dirs failed, the other 65 tests (including
ld_library_path_outranks_loader_cache_and_search_dirs) stayed green --
then reverted the swap and reran clean before committing.
Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
… first
HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.
The search stopped at the first match, so the second copy could not be
seen at all. It now records every copy found, in loader search order.
Attributing each copy to the installation that owns it, and doing the
same search for the HIP runtime so the two can be compared, is deferred;
comgr_matches_runtime stays permanently unset until that lands.
The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why `managed_sdk_ld_library_path` (the
environment the CLI builds to run an SDK tool, such as
rocm_agent_enumerator, inside a managed runtime) walks that directory.
Searching the root alone therefore missed the copy this CLI installs
itself, which is the case the entry exists for.
That knowledge now lives in one place for this search:
collect_managed_runtime_library_paths is shared by the comgr search here
and by managed_sdk_ld_library_path, so that one layout is described once
there rather than twice. A third description, collect_runtime_environment_paths,
still builds the environment a served engine process gets; unifying all
three is follow-up work, not part of this search.
Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.
The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record, root's own `_rocm_sdk_*` name is read
to find its parent -- the site-packages directory its siblings share --
and a miss costs a directory that is simply not reported.
No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.
Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.
The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.
Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.
The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.
Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.
Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.
Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.
Review: four more issues. (1) comgr_selected and the multi-copy note named
the ld.so.cache system copy as "would load" whenever a managed wheel
runtime was active but this process's own LD_LIBRARY_PATH was empty --
wrong, because rocm serve/rocm chat prepend the active runtime's own
library directories onto LD_LIBRARY_PATH before launching the engine, so
the wheel copy is what a served process actually loads. probe_comgr now
takes the active runtime's interpreter (the same FrameworkInterpreter
already threaded through probe_framework) and searches its library_paths
first, ahead of the ambient environment; pinned by
the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The
site_packages: None fallback in collect_managed_runtime_library_paths
guessed a venv layout (root/lib/<python>/site-packages) that the installer
never produces -- root is one of the runtime's own _rocm_sdk_* package
directories, not a venv root, so the guess always found nothing. It now
reads root.parent() instead, which is the site-packages directory the
guessed layout could never reach. The test fixture
(wheel_runtime_on_disk) was built from the same wrong venv shape and so
could not have caught this; rebuilt it from the real installer layout
(ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed
a_wheel_runtime_with_no_registry_record_is_still_found fails against the
old fallback before passing against the fix. (3) probe_comgr,
comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so
the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the
selection, and the multi-copy note could all break with the rest of the
suite still green. Split the ordering/selection logic into a pure
find_comgr_copies, the ldconfig -p line parsing into
parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings
(opt directory taken as a parameter rather than hardcoded, so a test can
point it at a temp folder instead of the real /opt on the machine running
the suite); each is unit-tested directly, and each new test was confirmed
to fail against a deliberately broken version of the function it covers
before being restored. (4) This message previously claimed a HIP copy
list and a single shared description of the loader path launched
processes use; neither matches this diff. Corrected above.
Review: a second round, one automated and one human, reaching the same
place by different routes.
The human reviewer confirmed, independently, that the site_packages:
None fallback added in the prior round could not actually fire:
ROCM_SDK_PROBE_SCRIPT has recorded site_packages unconditionally, outside
its try, since the probe's initial version, so every manifest that
parses at all carries Some there. The test built to cover it could not
have caught that either way, since it called the function directly with
None rather than through a real candidate. Rather than keep a reachable-
looking branch and a comment describing behaviour the code does not
have, the None arm is now a no-op (root's own directories, already
collected unconditionally above, are what a hand-constructed None caller
still gets) and the test is removed. The neighbouring root-format test
is kept and re-targeted: it already called the same function directly
with None, so it is what actually exercises that arm's contract, not the
removed test.
The automated reviewer caught a real regression the first round
introduced: three new unit tests feed a path into the `:`-split
LD_LIBRARY_PATH parser without a unix cfg gate, same as the pattern the
file already uses for that reason, and that broke the required
windows-build-and-test lane. Gated, along with the helper they share
(which would otherwise be unused, and clippy-denied, once its only
callers are gated).
It also found that the active-runtime-priority fix regresses
examine-finds-the-managed-runtimes-own-compilation-library on a healthy
host: once a managed runtime is active, its directory is found first and
labelled active-runtime, and the dedup this search has always done on
resolved path then drops the later managed-runtime hit for the identical
file -- the scenario's step asserted on the label that fix makes
unreachable. The step now accepts either label as proof the CLI's own
installed copy was found, and a new unit test
(an_active_runtime_directory_that_is_also_a_managed_root_keeps_one_label)
pins that an overlapping directory keeps exactly one label rather than
being reported twice; confirmed to fail when the active-runtime search is
moved after the others, then restored.
Review: a third round, three issues.
(1) comgr_matches_runtime was declared, defaulted, and listed in the
expected-keys test, but nothing in this diff computes it -- comparing
comgr's copies against the HIP runtime's is still deferred, same as the
commit message already said. A reader of the JSON contract alone had no
way to know the field was permanently unset rather than merely not yet
exercised by a test. Dropped from this commit entirely (the struct field,
its Default entry, its expected-keys assertion); `serde(default)` makes
re-adding it free once something actually computes it.
(2) comgr_selected named the first entry in comgr_paths as "would load"
regardless of which tier found it, so a machine carrying only a leftover
/opt/rocm-* install or an inactive managed runtime -- never a copy on
active-runtime, ld-library-path, or loader-cache -- was told that stale
copy would load, when the loader does not search /opt or a managed
runtime's directory on its own. find_comgr_copies now selects only from
those three loader-consulted tiers (LOADER_PATH_SOURCES), leaving
comgr_selected None and rewording the note when every hit is
rocm-install/managed-runtime-only; pinned by
a_search_dir_only_hit_is_not_reported_as_the_one_that_would_load. The e2e
step (assert_comgr_selection_is_stated) now asserts the same tier rule
against the real JSON: selected absent plus only non-loader-tier copies
is the pass case, not a shape to tolerate.
(3) comgr_search_dirs and managed_runtime_roots had no unit tests of
their own, and swapping the loader-cache and search-dir loops inside
find_comgr_copies left every then-existing test green -- nothing pinned
their relative order. Split comgr_search_dirs into a pure
comgr_search_dirs_in (rocm_path, the /opt directory, and the resolved
managed runtimes all taken as parameters) and added
comgr_search_dirs_labels_each_tier, covering all three source labels
against temp directories. Added loader_cache_outranks_search_dirs, built
so an ld-library-path hit cannot mask the order under test. Confirmed the
claim directly: swapped the two loops, reran the suite -- exactly
loader_cache_outranks_search_dirs failed, the other 65 tests (including
ld_library_path_outranks_loader_cache_and_search_dirs) stayed green --
then reverted the swap and reran clean before committing.
Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
… first (ROCm#384) * feat(examine): report every code object manager library, not just the first HIP compiles device code at run time through libamd_comgr, and a machine can hold more than one copy. The loader picks one. When the copy it picks does not belong to the active HIP runtime, compilation fails with an error naming neither the library nor the second copy. The search stopped at the first match, so the second copy could not be seen at all. It now records every copy found, in loader search order. Attributing each copy to the installation that owns it, and doing the same search for the HIP runtime so the two can be compared, is deferred; comgr_matches_runtime stays permanently unset until that lands. The managed search now reads the runtime registry rather than listing <data>/runtimes on disk. A directory listing yields a root and nothing else, and the root alone does not locate a wheel runtime's libraries -- those live in a sibling _rocm_sdk_* package inside the interpreter's site-packages, which is why `managed_sdk_ld_library_path` (the environment the CLI builds to run an SDK tool, such as rocm_agent_enumerator, inside a managed runtime) walks that directory. Searching the root alone therefore missed the copy this CLI installs itself, which is the case the entry exists for. That knowledge now lives in one place for this search: collect_managed_runtime_library_paths is shared by the comgr search here and by managed_sdk_ld_library_path, so that one layout is described once there rather than twice. A third description, collect_runtime_environment_paths, still builds the environment a served engine process gets; unifying all three is follow-up work, not part of this search. Listing the directory was also a second answer to "which runtimes exist", where the registry is the first one, and only the registry records which site-packages an interpreter uses. The record is preferred but not required. A read-only probe cannot depend on one being present and current, and a host whose registry is missing or stale is exactly the kind this command is called on: reporting no copies there would read as a machine with nothing wrong rather than one we failed to inspect. With no record, root's own `_rocm_sdk_*` name is read to find its parent -- the site-packages directory its siblings share -- and a miss costs a directory that is simply not reported. No dlopen. Reading the version through amd_comgr_get_version would run an unknown library's initialisers on a machine called on precisely because something is already wrong, and load a possibly conflicting HIP stack permanently into the process. The version is read from the versioned soname instead, and a name carrying none yields an empty version rather than a confident wrong one. Directory matches are now sorted, which makes the HIP probe's tie-break deterministic where one directory holds several matching files. It used to take whatever read_dir yielded first. Better, but a change, not a no-op. The loader-cache tier used to gate on which("ldconfig"), which only walks $PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian and derivatives. That silently dropped the one tier that finds a copy registered only in ld.so.cache; it now reuses the fallback search ldconfig_cache() already does for the same reason. Covered at two levels, because neither is sufficient alone. Unit tests build a wheel-format layout directly, which proves the search understands a layout we described; only a real managed runtime proves it matches the one the installer produces, so examine-19 asserts that on a GPU lane. The four new fields are `serde(default)`. This structure is read back from another machine: `rocm remote doctor` deserializes an examination the remote's own CLI produced, and that CLI may predate these fields. Without a default, adding one here refuses every remote running an older build, reported as "the remote CLI is probably a different version" -- true, and useless, since the older CLI is the one that cannot be changed. Fix CI: managed_runtime_roots resolved the data directory through crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default) instead of AppPaths::discover, so it disagreed with the rest of the CLI and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates the data directory -- which is exactly what every GPU e2e lane sets for isolation. That is why the new GPU scenario (examine-finds-the-managed-runtimes-own-compilation-library) failed identically on every GPU family; it now resolves through AppPaths::discover like everything else in the CLI. Also fixes a needless-collect clippy lint in the e2e step, and tags that scenario @requires-os:linux: the search looks only for libamd_comgr by its ELF soname, which native Windows does not ship. Rebasing onto main picked up two unrelated scenarios that claimed the same numbers in turn -- examine-16 (EAI-8950, ROCm#444), then examine-17 (EAI-8449, ROCm#434) -- so this commit's two new scenarios are renumbered to examine-18 and examine-19 to keep every scenario number in the file unique. Review: examine-18 (the host-independent code-object-manager scenario) was untagged and its empty-result assertion matched `Examination::default()` alone, so it would pass with `probe_comgr` deleted outright -- on native Windows, where the probe never runs, that is not a hypothetical. Tagged @requires-os:linux and the assertion now also requires the "no libamd_comgr found" note that only a probe which actually ran and came up empty pushes. Verified the new assertion fails by temporarily suppressing that note and watching the scenario go red, then restored it. Review: four more issues. (1) comgr_selected and the multi-copy note named the ld.so.cache system copy as "would load" whenever a managed wheel runtime was active but this process's own LD_LIBRARY_PATH was empty -- wrong, because rocm serve/rocm chat prepend the active runtime's own library directories onto LD_LIBRARY_PATH before launching the engine, so the wheel copy is what a served process actually loads. probe_comgr now takes the active runtime's interpreter (the same FrameworkInterpreter already threaded through probe_framework) and searches its library_paths first, ahead of the ambient environment; pinned by the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The site_packages: None fallback in collect_managed_runtime_library_paths guessed a venv layout (root/lib/<python>/site-packages) that the installer never produces -- root is one of the runtime's own _rocm_sdk_* package directories, not a venv root, so the guess always found nothing. It now reads root.parent() instead, which is the site-packages directory the guessed layout could never reach. The test fixture (wheel_runtime_on_disk) was built from the same wrong venv shape and so could not have caught this; rebuilt it from the real installer layout (ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed a_wheel_runtime_with_no_registry_record_is_still_found fails against the old fallback before passing against the fix. (3) probe_comgr, comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the selection, and the multi-copy note could all break with the rest of the suite still green. Split the ordering/selection logic into a pure find_comgr_copies, the ldconfig -p line parsing into parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings (opt directory taken as a parameter rather than hardcoded, so a test can point it at a temp folder instead of the real /opt on the machine running the suite); each is unit-tested directly, and each new test was confirmed to fail against a deliberately broken version of the function it covers before being restored. (4) This message previously claimed a HIP copy list and a single shared description of the loader path launched processes use; neither matches this diff. Corrected above. Review: a second round, one automated and one human, reaching the same place by different routes. The human reviewer confirmed, independently, that the site_packages: None fallback added in the prior round could not actually fire: ROCM_SDK_PROBE_SCRIPT has recorded site_packages unconditionally, outside its try, since the probe's initial version, so every manifest that parses at all carries Some there. The test built to cover it could not have caught that either way, since it called the function directly with None rather than through a real candidate. Rather than keep a reachable- looking branch and a comment describing behaviour the code does not have, the None arm is now a no-op (root's own directories, already collected unconditionally above, are what a hand-constructed None caller still gets) and the test is removed. The neighbouring root-format test is kept and re-targeted: it already called the same function directly with None, so it is what actually exercises that arm's contract, not the removed test. The automated reviewer caught a real regression the first round introduced: three new unit tests feed a path into the `:`-split LD_LIBRARY_PATH parser without a unix cfg gate, same as the pattern the file already uses for that reason, and that broke the required windows-build-and-test lane. Gated, along with the helper they share (which would otherwise be unused, and clippy-denied, once its only callers are gated). It also found that the active-runtime-priority fix regresses examine-finds-the-managed-runtimes-own-compilation-library on a healthy host: once a managed runtime is active, its directory is found first and labelled active-runtime, and the dedup this search has always done on resolved path then drops the later managed-runtime hit for the identical file -- the scenario's step asserted on the label that fix makes unreachable. The step now accepts either label as proof the CLI's own installed copy was found, and a new unit test (an_active_runtime_directory_that_is_also_a_managed_root_keeps_one_label) pins that an overlapping directory keeps exactly one label rather than being reported twice; confirmed to fail when the active-runtime search is moved after the others, then restored. Review: a third round, three issues. (1) comgr_matches_runtime was declared, defaulted, and listed in the expected-keys test, but nothing in this diff computes it -- comparing comgr's copies against the HIP runtime's is still deferred, same as the commit message already said. A reader of the JSON contract alone had no way to know the field was permanently unset rather than merely not yet exercised by a test. Dropped from this commit entirely (the struct field, its Default entry, its expected-keys assertion); `serde(default)` makes re-adding it free once something actually computes it. (2) comgr_selected named the first entry in comgr_paths as "would load" regardless of which tier found it, so a machine carrying only a leftover /opt/rocm-* install or an inactive managed runtime -- never a copy on active-runtime, ld-library-path, or loader-cache -- was told that stale copy would load, when the loader does not search /opt or a managed runtime's directory on its own. find_comgr_copies now selects only from those three loader-consulted tiers (LOADER_PATH_SOURCES), leaving comgr_selected None and rewording the note when every hit is rocm-install/managed-runtime-only; pinned by a_search_dir_only_hit_is_not_reported_as_the_one_that_would_load. The e2e step (assert_comgr_selection_is_stated) now asserts the same tier rule against the real JSON: selected absent plus only non-loader-tier copies is the pass case, not a shape to tolerate. (3) comgr_search_dirs and managed_runtime_roots had no unit tests of their own, and swapping the loader-cache and search-dir loops inside find_comgr_copies left every then-existing test green -- nothing pinned their relative order. Split comgr_search_dirs into a pure comgr_search_dirs_in (rocm_path, the /opt directory, and the resolved managed runtimes all taken as parameters) and added comgr_search_dirs_labels_each_tier, covering all three source labels against temp directories. Added loader_cache_outranks_search_dirs, built so an ld-library-path hit cannot mask the order under test. Confirmed the claim directly: swapped the two loops, reran the suite -- exactly loader_cache_outranks_search_dirs failed, the other 65 tests (including ld_library_path_outranks_loader_cache_and_search_dirs) stayed green -- then reverted the swap and reran clean before committing. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com> * feat(diagnose): report a code object manager that is not the runtime's own (ROCm#385) Builds on the copy search: with every copy of libamd_comgr known, the catalog can say when the one that loads belongs to a different installation than the HIP runtime that loads, and device code compilation therefore fails with an error naming neither. The rule is symmetric, and that is what makes it safe. The design note for this entry proposed a special case -- "when the active runtime is the managed runtime, the matching copy is the wheel copy" -- without which it "fires on every healthy CLI installation". That is a patch over a rule stated asymmetrically. Asking one question of both libraries instead, does the libamd_comgr that would load come from the same installation as the libamdhip64 that would load, makes every case fall out of the rule: a healthy managed install takes both from the managed runtime, so nothing differs and nothing is reported. A control that has to be written as an exception is a rule that has not been stated correctly yet. That turns on attributing a copy to its installation correctly. A managed runtime spreads its libraries across separate _rocm_sdk_* packages that sit beside each other under one site-packages, not nested under the runtime's own root at all, so ownership is decided by looking a found file's directory up against the same directories install_library_dirs already recorded for each root -- never by walking up from the file, which would call each package its own install and fire on the most common install we ship. That is the property the mutation check targets. Two further guards on firing: an unattributed copy is a gap in what the search knows rather than a finding about the machine, and the runtime must actually ship a copy of its own, or there is nothing to point the user at. The entry is print-only and ranks neither remedy. Removing a stack and reordering the search path can each break a working Python environment, and which is right depends on which stack the user means to keep. The finding says out loud that it describes the environment outside the managed runtimes. `rocm serve` puts a managed runtime's libraries first on purpose and gets a different, correct answer, and a report that did not name the environment it examined would read as a claim about one it never looked at. The rule above is now stated exactly once, as `comgr_matches_runtime` in examine.rs, and both this entry and `rocm examine --json`'s field of the same name call it rather than each restating the conditions -- so the two surfaces cannot disagree about one machine, and a regression test pins that. Fix option (b)'s advice matched an earlier, asymmetric statement of the rule; it now says to prefer the wheel's own copy of both libraries, which is what the symmetric rule actually asks the user to do. The machine-readable inspection now carries the same `install_root` attribution for the HIP runtime side that it already carried for the code object manager side, since the conflict question is about both libraries and a reader could not previously check either the attribution or the selection for one of them. Known roots are deduplicated across the whole list, not only where repeats happen to land next to each other. `dedup_by` drops consecutive duplicates, and these roots arrive from three independent sources -- the active install, a sorted /opt scan, and the managed runtimes -- so the active install colliding with a sibling was caught or missed depending on where the path sorted. A root recorded twice can make a machine holding one stack read as a machine holding two, which is the false conflict this entry exists not to report. hip_paths and hip_selected are `serde(default)` for the same reason the comgr fields are: an examination read back over the remote path may have been produced by an older CLI that never wrote them. Once a managed runtime is found, its root resolves to its `_rocm_sdk_devel` package directory, and the search used to guess a venv layout (`root/lib/<python>/site-packages`) to find its sibling packages; that guess does not hold for a real install, so the search now reads back the library directories the SDK probe already recorded (`library_paths`) instead of re-deriving them. This generalises the walker introduced in ROCm#384 (`find_library_copies`/`install_library_dirs`) so it serves the HIP side too, which is why `managed_runtime_roots` now returns `library_paths` plural rather than a single `site_packages` guess. The AppPaths::discover data-dir fix and the @requires-os:linux platform tags this commit originally carried for examine-17/18 belong with the scenario they correct, which ROCm#384 introduced -- moved there so the parent doesn't depend on the child to pass its own CI, and this commit now only rebases on top of that fix rather than restating it. Sweeping the same gap once it was visible: examine-17 (the host-independent code-object-manager scenario, renumbered by ROCm#384's rebase) asserts `hip_paths`/`hip_selected` the same way it asserts `comgr_paths`/`comgr_selected`, and the empty branch had the identical problem ROCm#384's review flagged for the comgr side -- `hip_paths: []` / `hip_selected: null` also hold by nothing more than `Examination`'s own defaults, so the assertion could not tell "probed, found none" from "never probed". `probe_comgr` now pushes a "no libamdhip64 found" note the same way it already did for libamd_comgr, and the HIP assertion now requires it in the empty branch. Verified by temporarily suppressing the note and watching the scenario go red, then restoring it. Review: the attribution half still did not survive a realistic managed runtime. `known_install_roots` registers a managed runtime by `candidate.root_path` alone -- `_rocm_sdk_devel` when the `devel` extra is installed -- and `libamd_comgr`/`libamdhip64` live in the sibling `_rocm_sdk_core` package, which sits *beside* `root_path`, not under it. A string-prefix match against the bare root therefore attributed nothing on exactly this shape: both libraries came back `install_root: ""`, `comgr_matches_runtime` hit its empty-root guard and returned `None` instead of `true`, and `check_18_comgr_conflict` could not fire for that runtime at all. The two unit tests meant to catch this (`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime`, `a_healthy_managed_installation_raises_no_report`) built their fixtures from a venv root (`/data/runtimes/therock/default`, `.../lib/python3.12/site-packages/_rocm_sdk_core/...`) that production never produces, so they passed regardless. Fixed at the root: `install_library_dirs` already walks, for every known root, exactly the directories that root's copies live in (that is the discovery-side fix this commit already made). `find_library_copies` now builds a directory -> owning-root map from that same walk and attributes each found file by looking its directory up in it, rather than by string-prefix match against the bare root. A sibling package's directory is simply one more entry in the map, keyed to the runtime that recorded it, so it attributes correctly without being nested under anything. `owning_install_root` and `record_library_copy` take that map instead of the bare root list; `the_nearest_enclosing_installation_claims_a_library` is removed because the "longest prefix wins" ambiguity it guarded against cannot arise under exact directory lookup. Both fixture tests above were rebuilt from the real installer shape (sibling `_rocm_sdk_*` packages under one `site-packages`, matching `apps/rocm/src/therock.rs`'s `ROCM_SDK_PROBE_SCRIPT`), and `a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` was confirmed to fail against the old string-prefix attribution before passing against the fix. `find_library_copies` also gained the `active_runtime_dirs` parameter runtime's own library directories are now searched first, ahead of the ambient `LD_LIBRARY_PATH`, for both comgr and HIP, since `rocm serve` prepends them for both libraries equally. The loader cache and the directory walk are now gathered once in `probe_comgr` and passed to both the comgr and HIP searches, rather than each search reading `LD_LIBRARY_PATH`, spawning `ldconfig -p`, and walking every managed runtime's directories again on every invocation -- neither depends on which library is being searched for, so the duplication cost an extra subprocess and directory walk on every `rocm examine`/`rocm diagnose` for no reason. `multi_copy_note` is split out as its own pure function so the "N copies ... would load" note stays independently testable now that `find_library_copies` only returns copies. Two doc-comment splices, both from a missing blank `///` line between two unrelated items: `check_18_comgr_conflict`'s doc paragraph had absorbed check_17's above it, leaving check_17 undocumented; the same shape left `known_install_roots` undocumented under a paragraph describing `dedup_roots_keeping_first`. Both are split back onto the function they actually describe. `docs/wsl.md` said `fix-19-shm-too-small` was the only non-WSL entry that applies on WSL; registering `check_18_comgr_conflict` for `["linux", "wsl"]` made that false the moment this entry existed. Updated to name both. The "Neither option is recommended..." sentence was stated independently in `fix.rs`'s static recipe and `diagnose.rs`'s dynamically-generated one, in slightly different words -- already drifted, and only the `fix.rs` copy was pinned by an e2e assertion, so the `diagnose.rs` copy could drift further with nothing to catch it. Both now read `fix::COMGR_CONFLICT_NEITHER_OPTION_RECOMMENDED`, so they cannot disagree. Not changed: `fix.rs`'s option (b) wording and `comgr_matches_runtime`'s parity with `check_18_comgr_conflict` were flagged in an earlier review round and were already correct by the time this round started -- verified against the current code rather than re-fixed. Rebase onto ROCm#384 and a second symmetry gap: ROCm#384's own review fixed `comgr_selected` picking `copies.first()` regardless of tier -- a `rocm-install`/`managed-runtime`-only hit used to be reported as "the one that would load" purely because it sorted first, when neither source is one the loader consults on its own. Rebasing that fix under this entry's HIP-runtime work surfaced a second instance of the same asymmetry a human reviewer caught directly on this branch: `probe_comgr`'s doc comment claimed comgr and HIP were searched and reported identically, but only comgr ever got a `multi_copy_note` call; the HIP branch only checked `hip.is_empty()` and pushed nothing for the "found, but only in a directory the loader does not consult" case the comgr side had just been fixed to report. Fixed by stating the tier rule once and applying it to both prefixes rather than fixing it twice: `select_loader_copy` replaces `copies.first()` with a search restricted to `LOADER_PATH_SOURCES` (`active-runtime`, `ld-library-path`, `loader-cache` -- the tiers the loader itself consults), and `copies_note` replaces `multi_copy_note` with the three-way note (no copies / more than one with a winner / one-or-more with none selected) that a `rocm-install`-only hit now needs. `probe_comgr` calls both exactly once for comgr and once for HIP, so there is one call site per library rather than one function's logic duplicated by hand into the other's branch, and the doc comment now says what the code does instead of what it used to do before ROCm#384's fix landed. Pinned by `a_search_dir_only_hit_is_not_selected_for_either_prefix`, which runs the same assertions for `COMGR_LIB_PREFIX` and `HIP_RUNTIME_LIB_PREFIX` in one test rather than two, so the rule is proven to apply to both rather than merely written to apply to both. The existing `loader_cache_outranks_search_dirs` and `the_active_runtimes_own_copy_outranks_the_ambient_environment`/ `ld_library_path_outranks_loader_cache_and_search_dirs` tests were carried through the rebase onto `find_library_copies`/ `select_loader_copy` rather than dropped, since they pin mutation-tested tier-ordering bugs ROCm#384 already fixed once. `comgr_search_dirs_labels_each_tier`, which tested the now-deleted `comgr_search_dirs_in`, is removed as redundant: `install_library_dirs`'s own tier-labelling is already covered by `a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` and `a_library_no_installation_claims_is_attributed_to_none`. Verified: `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo clippy -p e2e-cucumber --test e2e -- -D warnings`, and `cargo test --workspace --all-targets` all pass (one `xtask::workflow_contract` test flaked under full-suite parallel load on a timing assertion against a real `rocm-smi` subprocess call and passed cleanly in isolation and on a full-suite rerun; unrelated to this change). Cross-compiled the whole workspace, tests included, for `x86_64-pc-windows-gnu` under `-D warnings` to confirm no code is left unix-only-reachable and therefore dead on Windows. `cargo xtask e2e -- -n skill` passes all 6 scenarios, since this round also touches `skills/rocm-doctor/reference.md`. Second review round, two independent reviewers filing the same four findings (verified rather than merely accepted): fix-18's evidence line claimed to describe "the environment as it stands outside the CLI's managed runtimes" unconditionally, even when a selection actually came from the active-runtime tier -- a served child's search order, not a plain shell's. That also means the `export LD_LIBRARY_PATH=...` remedy can be inert for exactly that case, since `lemonade_process_environment_vars` prepends the runtime's own directories ahead of whatever the remedy exports. Rewording the evidence line is the easy half; the detection logic that decides which sentence to print cannot move to plain-shell-only without reintroducing the false positives/negatives `comgr_matches_runtime` and `a_healthy_managed_installation_raises_no_report` were written to rule out, since that guarantee depends on both selections already being active-runtime-tier-aware. So `comgr_matches_runtime` is untouched; `check_18_comgr_conflict` now checks whether either selected copy's source is `active-runtime` and, when it is, swaps in an honest sentence plus a new remedy note stating the export may not reach a served child and pointing at reinstalling/repairing that runtime instead. Pinned by `an_active_runtime_s_own_hip_without_its_own_comgr_notes_the_scope_limit`, mutation-tested by forcing the branch condition false and watching the new test fail on the stale sentence, then reverting. `assert_hip_selection_is_stated`, the e2e HIP twin of the comgr selection assertion, had only two branches, so "copies found, but none on a tier the loader consults" -- `select_loader_copy` returning `None` on purpose, already pinned for `libamdhip64` by `a_search_dir_only_hit_is_not_selected_for_either_prefix` -- fell into the `else` arm and asserted against a selected copy that does not exist. Added the missing third branch, mirroring `assert_comgr_selection_is_stated` exactly, including its `LOADER_PATH_SOURCES` check on the selected copy's source, which the HIP step's final arm was also missing. Hoisted the shared `LOADER_PATH_SOURCES` const the two steps now both use so they cannot drift apart again. `skills/rocm-doctor/reference.md`'s "Closed catalog (24 failure modes)" heading and `SKILL.md`'s "the other 20 are print-only" line had both gone stale against the table's 25 rows and 4 auto-applicable fixes (`exactly_the_four_known_fixes_are_auto`); fixed to 25 and 21. `rocm_doctor_skill.feature` deliberately leaves both counts unparsed against the live CLI, and that scope decision stands -- the CLI is not wired to report "how many rows exist" as a fact an agent reads. What it does not rule out is checking the two numbers against each other, so `tests/e2e-cucumber/tests/skill_reference.rs` (plain `cargo test`, no binary needed) gained `free_standing_catalog_counts_match_the_table`, which parses both numbers back out of the two docs and asserts them against the table's own row and auto-fix counts. Mutation-tested by reverting each number one at a time and confirming the new test goes red, then restoring it. Open question from the first round, never answered: does `install_library_dirs` calling `collect_sdk_library_paths` unconditionally widen what a bare-metal `/opt/rocm` root probes in a way that matters? Traced it rather than taking the earlier answer on faith: every directory `install_library_dirs` adds for a root carries that root's own source (`rocm-install` or `managed-runtime`, never a loader-consulted tier), and `find_library_copies` only records a copy under that source when no loader-consulted tier (`active-runtime`/`ld-library-path`/`loader-cache`) already found the same path first -- so a wider probe can only add more `rocm-install`-sourced evidence, never promote a path onto a tier `select_loader_copy` would pick. The one place a wider probe does change an answer is `comgr_matches_runtime`'s "does the runtime ship its own copy" check, and there a new hit can only turn a false conflict report into `None` (not enough evidence), which is strictly more correct, not a new risk. The reviewer's reasoning holds; recorded here since the first round left it unanswered. Re-verified after this round's changes: `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo clippy -p e2e-cucumber --test e2e -- -D warnings`, `cargo test --workspace --all-targets` (251 passed, 0 failed across 33 binaries), and `RUSTFLAGS="-D warnings" cargo check --workspace --all-targets --target x86_64-pc-windows-gnu` all pass clean. `cargo xtask e2e -- -n skill` passes all 6 scenarios again; `cargo xtask e2e -- -n examine-18` passes its 5 steps, confirming the new HIP e2e branch does not break the existing scenario. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com> --------- Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
The rocminfo walker kept a running agent name that every `Name:` line overwrote, and a GPU agent lists its ISAs as further `Name:` lines (`amdgcn-amd-amdhsa--gfx1100`, `...--gfx11-generic`) after `Device Type:`. By the next `Agent` header the name no longer started with `gfx`, so every GPU agent was dropped and rocminfo contributed nothing: no marketing name for a card the PCI scan could not name, no `is_apu` from the target's family. The per-GPU `gfx_target` that was empty when ROCm#393 was filed is now filled by the kernel membership pass (ROCm#434); this fixes the parser so the rest of rocminfo's reading reaches the report. Take the first `Name:` after an `Agent` header as the agent's name and ignore the rest, same for `Marketing Name:` and `BDFID:`. The loop is lifted into `parse_rocminfo_gpu_agents`, and `apply_rocminfo_gpu_agents` joins agents to entries by PCI address (the `BDFID` rendered in the `dddd:bb:dd.f` form `pci_ids_match` already compares) instead of by position, so rocminfo's and lspci's enumeration orders need not agree. Only an agent with no usable `BDFID:` falls back to position, a fallback never displaces an exact match, and an agent whose `BDFID:` matched nothing becomes its own GPU rather than land on an unrelated card. rocminfo never overwrites a `gfx_target` already present. The membership pass stamps each entry with the kernel's native target by address before rocminfo runs; rocminfo's reading is the same value normally and the masqueraded one under `HSA_OVERRIDE_GFX_VERSION`, which the probe also scrubs from rocminfo's environment so an override cannot be read back as the hardware's gfx. `is_apu` follows the target's family wherever the family table has a verdict (`gfx_apu_family_verdict`: known APU, known discrete, or not covered) and keeps the earlier name-based guess otherwise; the hipInfo path on Windows uses the same rule. A Raphael iGPU (gfx1036, not in the table) keeps lspci's APU verdict instead of flipping to discrete, and a GPU appended with no entry to fill is left `None` when its family is unknown. `rocm diagnose`'s two framework-arch checks read every AMD target. Now that ROCm#434 fills the iGPU's target, an APU+dGPU desktop on upstream PyTorch wheels is told to reinstall torch for a device fix-9 tells it to hide, and the mirror check goes quiet. `serving_gfx_targets` judges the discrete targets only when a host has both kinds of GPU, naming the integrated target it left out, and falls back to every target when no discrete one is known. Hosts with one kind are judged as before. No new scenario: examine-17 already asserts every GPU's PCI address and gfx target against the kernel topology, and the diagnose change only diverges on an iGPU+dGPU host whose wheel misses the iGPU, which no CI lane provides; that shape is covered by unit tests. Fixes ROCm#393 Signed-off-by: Shubham Thakur <sbmthakur@gmail.com>
Summary
rocm examinecould not see AMD Instinct GPUs through itslspciprobe. Instinct parts enumerate under PCI class 1200 ("Processing accelerators"), not a display class, butprobe_gpus_lspcimatched onlyVGA compatible controller,3D controllerandDisplay controller. Every datacenter GPU was therefore skipped, leavinghas_amd_gputo rest entirely onrocminfoor the sysfs fallback — and reportfalsewhen neither was reachable (EAI-8449).Verified firsthand on an 8-GPU MI300X host, where all eight devices read
class=0x120000in sysfs and reallspci -nn -Drenders them as:On that host, before and after:
has_amd_gpugpus[]countgpus[].pci_id""has_discrete_amdgpus[].gfx_targetNote on the original report
The upstream report attributed this to GPU enumeration being gated on a legacy
amdkfdkernel module. That mechanism does not exist in the code, and the hardware confirms it: the MI300X host reportsamdkfd_loaded: falseandhas_amd_gpu: truein the same run (on modern kernels amdkfd is built intoamdgpu, so it never appears inlsmod). The symptom was real; the stated cause was not.What changed
is_lspci_gpu_line— the class filter, extracted fromprobe_gpus_lspciand widened to includeProcessing accelerators.kfd_gpu_nodes_in+apply_kernel_gpu_membership— count from the kernel, name from PCI. Widening the class filter also changed who decides which GPUs exist, and the PCI bus answers a different question from the kernel. In a container the host bus still carries every card the node has while KFD describes only the one passed through, soexamine --jsonlisted eight MI300X GPUs where one was usable. examine-16 caught exactly that on four lanes — "the kernel describes 1 GPU node(s) but the report lists 8 AMD GPU(s)" — and it was right:rocm servewould have failed on a GPUexaminehad just advertised, which is worse than the under-detection this change exists to fix, because that at least failed honestly.So
gpus[]is the set the kernel exposes, and the PCI scan enriches it.kfd_gpu_nodes_inreads the KFD topology per node and returns each node's own gfx target alongside the PCI address it reports;apply_kernel_gpu_membershipmatches those against whatlspcifound by address rather than by position or count. A matched node keeps lspci's marketing name and APU/discrete verdict and fills only an absent target. A node the PCI scan did not list is still reported — it is usable, and dropping it would under-report in precisely the way this change exists to fix — just without the enriched name. Devices on the bus that the kernel does not expose leavegpus[]but not the report: they are named in a note, because "the bus has it and the kernel does not" is itself a diagnosis.Two deliberate non-changes: an empty topology is a no-op rather than "no GPUs", so a host whose
amdgpunever bound keeps its PCI list and the driver probes explain why nothing is usable; and onlyis_amdentries are touched, since KFD says nothing about an NVIDIA card.probe_gpus_after_lspci— one ordered stage. The four post-lspci passes are gathered into a single function behind an injectableGpuProbeSourcesseam, so the ordering the fix depends on is pinned by assertion rather than by prose in a doc comment.rocminfo's name beats the kernel-topology placeholder, and the unknown-name note is taken between therocminfofold and the sysfs fallback.Test plan
lspci_matches_instinct_processing_accelerator_class— fails before the class fix, passes after (verified by reverting the predicate).the_unknown_name_note_is_taken_between_rocminfo_and_the_sysfs_fallback— drives the realprobe_gpus_after_lspcisequence through theGpuProbeSourcesseam, so a reordering of the post-lspci passes reddens a test instead of passing silently.the_report_lists_the_gpus_the_kernel_exposes_not_every_card_on_the_bus,a_kernel_visible_gpu_the_pci_scan_missed_is_still_listed,a_kernel_that_exposes_no_gpu_leaves_the_pci_enumeration_standing(empty topology is a no-op, not "no GPUs"),the_membership_read_reconciles_a_planted_topology_end_to_end,the_membership_probe_reconciles_the_topology_it_is_pointed_atandthe_membership_probe_leaves_the_report_alone_when_the_topology_is_unreadable.a_rocminfo_marketing_name_outranks_the_kernel_topology_placeholder— pins the precedence5d7c3cd4establishes over theAMD GPU (from kernel topology)placeholder.cargo test --workspace --all-targets,cargo clippy --workspace --all-targets -- -D warnings,cargo fmt --all -- --check— all pass, run on the MI300X host after rebasing onto currentmain.scripts/smoke_local.pypasses.rocm examinere-verified on MI300X after the guard was added: still 8 GPUs, allgfx942, all carrying PCI addresses.@id:examine-lists-every-gpu-with-its-address-and-target(examine-16) has not been executed. It is@requires-gpu, and the suite has no per-scenario filter, so it needs the GPU e2e lane. Verified by compilation and by thefeature_namingconvention tests only.On the e2e scenario
Per AGENTS.md §3 this change needs a Gherkin scenario, and no existing one covers it.
@id:examine-detects-gpu-and-driver(examine-04) assertsdetected_gfx_targetagainst KFD, and@id:examine-both-forms-agree-on-gpu(examine-08) comparesdetected_gfx_targetagainsthas_amd_gpu. Neither field moves here, andhas_amd_gpuwas alreadytruepre-fix via the sysfs fallback — so both pass identically before and after. Nothing in the suite asserted ongpus[],pci_id, orhas_discrete_amd.examine-16 reads
gpus[]directly, asserting one AMD entry per kernel GPU node, each carrying both its PCI address and its gfx target. (It no longer asserts count equality between the bus and the kernel — that was the earlier, superseded design;gpus[]is now the kernel's set, matched to lspci by address.) It no-ops where a premise cannot hold — no readable KFD topology, nolspcito supply PCI addresses, or a topology whose nodes disagree on a target — because on such a host the answer it would otherwise flag is the correct one.Checklist
Searched
expectations.tomlfor the fixed ticket ID: no EAI-8449 rows, nothing to remove or narrow.Related: the
EAI-7998xfail onexamine-both-forms-agree-on-gpuis scoped to WSL, and its comment notes the ticket "also records the same disagreement on MI300X, second-hand from the comment incapability.rs; if that is confirmed the row should widen". This work checked that firsthand — the two forms agree on MI300X (has_amd_gpu: true, human namesgfx942). The row should stay WSL-only and must not widen.No new subcommand or subsystem; architecture rule not applicable.
Scope caveat
The verification host is a container, so
lspci/rocminfowere not installed by default;pciutilswas installed there to exercise the lspci path end to end. The PCI class evidence itself came from sysfs and is independent of that. A bare-metal Instinct host with ROCm installed takes therocminfopath and was not available for testing.Fixes EAI-8449.