Skip to content

fix(examine): enumerate Instinct GPUs from lspci's accelerator class - #434

Merged
r0x0r merged 10 commits into
mainfrom
eai-8449-lspci-processing-accelerators
Oct 1, 2026
Merged

r0x0r merged 10 commits into
mainfrom
eai-8449-lspci-processing-accelerators

Conversation

@r0x0r

@r0x0r r0x0r commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

rocm examine could not see AMD Instinct GPUs through its lspci probe. Instinct parts enumerate under PCI class 1200 ("Processing accelerators"), not a display class, but probe_gpus_lspci matched only VGA compatible controller, 3D controller and Display controller. Every datacenter GPU was therefore skipped, leaving has_amd_gpu to rest entirely on rocminfo or the sysfs fallback — and report false when neither was reachable (EAI-8449).

Verified firsthand on an 8-GPU MI300X host, where all eight devices read class=0x120000 in sysfs and real lspci -nn -D renders them as:

0000:11:00.0 Processing accelerators [1200]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:74a1]

On that host, before and after:

before after
has_amd_gpu true true
gpus[] count 1 8
gpus[].pci_id "" real BDFs
has_discrete_amd false true
gpus[].gfx_target gfx942 gfx942

Note on the original report

The upstream report attributed this to GPU enumeration being gated on a legacy amdkfd kernel module. That mechanism does not exist in the code, and the hardware confirms it: the MI300X host reports amdkfd_loaded: false and has_amd_gpu: true in the same run (on modern kernels amdkfd is built into amdgpu, so it never appears in lsmod). The symptom was real; the stated cause was not.

What changed

  1. is_lspci_gpu_line — the class filter, extracted from probe_gpus_lspci and widened to include Processing accelerators.

  2. kfd_gpu_nodes_in + apply_kernel_gpu_membership — count from the kernel, name from PCI. Widening the class filter also changed who decides which GPUs exist, and the PCI bus answers a different question from the kernel. In a container the host bus still carries every card the node has while KFD describes only the one passed through, so examine --json listed eight MI300X GPUs where one was usable. examine-16 caught exactly that on four lanes — "the kernel describes 1 GPU node(s) but the report lists 8 AMD GPU(s)" — and it was right: rocm serve would have failed on a GPU examine had just advertised, which is worse than the under-detection this change exists to fix, because that at least failed honestly.

    So gpus[] is the set the kernel exposes, and the PCI scan enriches it. kfd_gpu_nodes_in reads the KFD topology per node and returns each node's own gfx target alongside the PCI address it reports; apply_kernel_gpu_membership matches those against what lspci found by address rather than by position or count. A matched node keeps lspci's marketing name and APU/discrete verdict and fills only an absent target. A node the PCI scan did not list is still reported — it is usable, and dropping it would under-report in precisely the way this change exists to fix — just without the enriched name. Devices on the bus that the kernel does not expose leave gpus[] but not the report: they are named in a note, because "the bus has it and the kernel does not" is itself a diagnosis.

    Two deliberate non-changes: an empty topology is a no-op rather than "no GPUs", so a host whose amdgpu never bound keeps its PCI list and the driver probes explain why nothing is usable; and only is_amd entries are touched, since KFD says nothing about an NVIDIA card.

  3. probe_gpus_after_lspci — one ordered stage. The four post-lspci passes are gathered into a single function behind an injectable GpuProbeSources seam, so the ordering the fix depends on is pinned by assertion rather than by prose in a doc comment. rocminfo's name beats the kernel-topology placeholder, and the unknown-name note is taken between the rocminfo fold and the sysfs fallback.

Superseded during review: an earlier round of this PR filled the gfx target with uniform_kfd_gfx_target_in plus a count-equality guard (every KFD node agreeing on one target, and exactly as many nodes as AMD GPUs enumerated). 407d8fc6 replaced that wholesale with the per-node, address-matched approach above, which does not need the guard because it never attributes one node's target to another device. Neither uniform_kfd_gfx_target_in nor its two tests exist at this head.

Test plan

  • Regression test lspci_matches_instinct_processing_accelerator_class — fails before the class fix, passes after (verified by reverting the predicate).
  • the_unknown_name_note_is_taken_between_rocminfo_and_the_sysfs_fallback — drives the real probe_gpus_after_lspci sequence through the GpuProbeSources seam, so a reordering of the post-lspci passes reddens a test instead of passing silently.
  • Kernel-membership coverage, all planted-topology: the_report_lists_the_gpus_the_kernel_exposes_not_every_card_on_the_bus, a_kernel_visible_gpu_the_pci_scan_missed_is_still_listed, a_kernel_that_exposes_no_gpu_leaves_the_pci_enumeration_standing (empty topology is a no-op, not "no GPUs"), the_membership_read_reconciles_a_planted_topology_end_to_end, the_membership_probe_reconciles_the_topology_it_is_pointed_at and the_membership_probe_leaves_the_report_alone_when_the_topology_is_unreadable.
  • a_rocminfo_marketing_name_outranks_the_kernel_topology_placeholder — pins the precedence 5d7c3cd4 establishes over the AMD GPU (from kernel topology) placeholder.
  • cargo test --workspace --all-targets, cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all -- --check — all pass, run on the MI300X host after rebasing onto current main.
  • scripts/smoke_local.py passes.
  • rocm examine re-verified on MI300X after the guard was added: still 8 GPUs, all gfx942, all carrying PCI addresses.
  • @id:examine-lists-every-gpu-with-its-address-and-target (examine-16) has not been executed. It is @requires-gpu, and the suite has no per-scenario filter, so it needs the GPU e2e lane. Verified by compilation and by the feature_naming convention tests only.

On the e2e scenario

Per AGENTS.md §3 this change needs a Gherkin scenario, and no existing one covers it. @id:examine-detects-gpu-and-driver (examine-04) asserts detected_gfx_target against KFD, and @id:examine-both-forms-agree-on-gpu (examine-08) compares detected_gfx_target against has_amd_gpu. Neither field moves here, and has_amd_gpu was already true pre-fix via the sysfs fallback — so both pass identically before and after. Nothing in the suite asserted on gpus[], pci_id, or has_discrete_amd.

examine-16 reads gpus[] directly, asserting one AMD entry per kernel GPU node, each carrying both its PCI address and its gfx target. (It no longer asserts count equality between the bus and the kernel — that was the earlier, superseded design; gpus[] is now the kernel's set, matched to lspci by address.) It no-ops where a premise cannot hold — no readable KFD topology, no lspci to supply PCI addresses, or a topology whose nodes disagree on a target — because on such a host the answer it would otherwise flag is the correct one.

Checklist

  • Searched expectations.toml for the fixed ticket ID: no EAI-8449 rows, nothing to remove or narrow.

    Related: the EAI-7998 xfail on examine-both-forms-agree-on-gpu is scoped to WSL, and its comment notes the ticket "also records the same disagreement on MI300X, second-hand from the comment in capability.rs; if that is confirmed the row should widen". This work checked that firsthand — the two forms agree on MI300X (has_amd_gpu: true, human names gfx942). The row should stay WSL-only and must not widen.

  • No new subcommand or subsystem; architecture rule not applicable.

Scope caveat

The verification host is a container, so lspci/rocminfo were not installed by default; pciutils was installed there to exercise the lspci path end to end. The PCI class evidence itself came from sysfs and is independent of that. A bare-metal Instinct host with ROCm installed takes the rocminfo path and was not available for testing.

Fixes EAI-8449.

…ator class

`probe_gpus_lspci` matched only the three display classes, but Instinct
parts carry no display class at all: an MI300X enumerates under PCI class
1200, "Processing accelerators". Every datacenter GPU was therefore
invisible to the lspci probe, leaving `has_amd_gpu` to rest entirely on
`rocminfo` or the sysfs fallback -- and report false when neither was
reachable, which is the reported symptom in EAI-8449.

Verified on an 8-GPU MI300X host, where all eight devices read
`class=0x120000` in sysfs and `lspci -nn -D` renders them as
`Processing accelerators [1200]`. Before: one topology-sourced entry with
no PCI id and `has_discrete_amd: false`. After: eight entries carrying
their real PCI ids, correctly classified as discrete.

Picking the richer lspci answer would otherwise have cost the gfx target,
since lspci does not know one and `rocminfo` was absent: the entries went
to an empty `gfx_target` where the fallback had supplied `gfx942`, and
`gpus[].gfx_target` is the only place the JSON report names a target at
all. So the fallback now also fills that gap from the kernel topology,
under the same rule it already followed for enumeration -- fill a gap,
never overwrite a target `rocminfo` resolved per-device.

Note this does not change the original report's root cause as filed: the
`amdkfd` module gating described there does not exist in the code, and the
MI300X host confirms it, reporting `amdkfd_loaded: false` alongside
`has_amd_gpu: true`.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Review of the previous commit found the gap-fill it added was unsafe on a
mixed host. `detect_kfd_gfx_target_in` deliberately returns the
lowest-numbered KFD node -- the right answer for HIP ordinal 0, and the
wrong one to pin on a *particular* device. On an APU+dGPU box without
`rocminfo`, `lspci` resolves the integrated part by name but leaves the
discrete card empty, so the fill stamped the APU's target onto the dGPU.

That is not inert: `diagnose`'s iGPU/dGPU check splits on `gfx_target` to
tell the user which GPU to pin, and the comment above it says naming the
wrong one is "actively wrong for exactly the APU+dGPU pairing this check
fires on". Before the fill the discrete entry was empty and the check took
its honest branch; after, it named a confident wrong target.

So read the topology differently for this purpose: `uniform_kfd_gfx_target_in`
reports a target only when every KFD GPU node agrees on one, together with
how many nodes it saw, and the fill additionally requires that count to
equal the number of AMD GPUs enumerated. A topology that does not account
for every card stays silent rather than guessing. The MI300X case that
motivated the fill is unaffected -- eight nodes, one target, eight GPUs --
and is re-verified on that hardware.

Also from review:

- Add the e2e scenario AGENTS.md requires for observable behavior. The two
  scenarios the previous commit relied on do not cover it: examine-04 and
  examine-08 assert `detected_gfx_target` and `has_amd_gpu`, neither of
  which the enumeration change moves, so both passed before and after.
  examine-16 reads `gpus[]` itself, cross-checked against the kernel's GPU
  node count, and no-ops where the premise cannot hold.
- Correct a doc comment claiming `gpus[].gfx_target` is the only place the
  JSON names a target; `framework_arch_list` and `summary` carry targets
  too. The true claim is narrower: the hardware's own target per device.
- Describe the fallback's second responsibility in its doc comment, which
  still said the function only discovers GPUs when nothing else could.
- Stop two comments in the same commit contradicting each other about
  whether lspci names an Instinct part "Device" -- it depends on the host's
  `pci.ids` vintage.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r
r0x0r requested a review from a team as a code owner September 24, 2026 07:21
@r0x0r
r0x0r requested a review from siloteemu September 24, 2026 07:21
@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 6699ec3

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Two commits teach the bus-listing probe to enumerate datacenter GPUs that carry a processing-accelerator class instead of a display class, and add a narrowly-guarded fill that copies a gfx target from the kernel topology onto enumerated GPUs only when the topology accounts for every one of them. Outcome: Needs work — the enumeration half is well-built and well-tested, but the gap-fill half has no test at any level. Verified: the merge base equals the base tip and both diffs produce an identical four-file set; the four new unit tests compile and pass locally (4 passed, 0 failed); I confirmed by reading source that the sorted node ordering makes the topology test deterministic rather than flaky, that non-AMD accelerators matched by the new class rule are dropped by the existing vendor gate so they cannot corrupt the counts, that the count guard can only ever suppress a fill and never produce a wrong one, and that the commit's claim about the two pre-existing scenarios not covering the enumeration change is accurate; a leak scan over the diff returned no hits and no prompt-injection attempt was found in code, comments, commit messages or feature text. The full suite, clippy and the e2e lanes were not run here. This review worked from a CI rollup of 23 success, 1 skipped, 4 failure; I could not attribute any red check to a specific defect — the one candidate is the new scenario's strict count equality (non-blocking item 3), which can fail on a legitimate host, but that is inference and I cannot confirm it. Blocking: 1 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

crates/rocm-core/src/examine.rs:1256-1258 — the gfx-target gap-fill, which is the entire second half of this PR, is wired in by a single statement that no test exercises in a state where it can do anything. Delete that line and the whole suite still passes. The two new unit tests call the inner helper directly with hand-supplied arguments, so they prove the helper and nothing about the wiring; the one pre-existing test that now reaches this branch (the_sysfs_fallback_leaves_a_real_pci_enumeration_alone) arrives with a GPU whose target is already populated, so the fill is a no-op there and its assertions cannot move either way; and the new scenario asserts entry count and PCI address but never reads gpus[].gfx_target. That field is user-observable output of the machine-readable report, and the contributor rules explicitly state that a unit test on the internal helper does not discharge the scenario requirement — "it proves the function, not the behavior". Concrete fix: extend the new step (or add one) to assert that every AMD entry carries a non-empty target whenever the kernel topology is uniform and its GPU-node count equals the number of AMD entries reported, no-opping on the same premise-failure conditions the step already uses; and give the fill a seam so a unit test can drive the early-return branch against a planted topology instead of the host's, the way the topology reader itself is already split.

Non-blocking

  • examine.rs:1078-1083 — the class match is on the English class name, but the invocation already passes the flag that guarantees the numeric class code is printed in brackets on every line; matching that code instead is free and immune to the database vintage the PR's own test comment reasons about for device names. If a host's ID database lacks the class entry the line renders with a generic placeholder and the fix silently stops firing on exactly the hosts it targets. I could not execute the listing tool here, so the exact placeholder rendering is inferred, not verified.
  • examine.rs:3045 — the new matcher test feeds only well-formed lines. It never feeds an empty line, a truncated line, or a line where the class name failed to resolve, so it cannot catch the failure mode above.
  • examine_steps.rs:843 — asserting exact equality between AMD entries and kernel GPU nodes is stricter than the behaviour under test. An AMD GPU the bus listing sees but the kernel does not register as a compute node (driver unbound or blacklisted, a legacy card), or an AMD-vendor non-GPU accelerator newly matched by the class rule, makes the counts legitimately differ and fails the scenario although the output is correct. Consider asserting "at least as many, each with an address" or excluding entries with no corresponding node.
  • lib.rs:5127-5141 — no test covers the zero-GPU-node or empty-directory path. If the short-circuit on the first element were replaced by something that does not short-circuit, the function would return a count of zero with an empty target and neither new case would notice.
  • examine.rs:1256-1258 — a side effect worth flagging on its own: the new call makes the pre-existing fallback test read the host's real kernel topology, which is precisely the machine dependence the neighbouring doc comment says the split-out seam exists to avoid. Harmless today because that test's assertions cannot be moved by a fill, but it quietly undermines the stated invariant and will mislead the next reader.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 6699ec3

Blocking: 1. This is the first round of review from this account on this pull request, and it answers the review request.

crates/rocm-core/src/examine.rs:1256-1258 - the gap-fill half of this change is wired in by a single statement that no test exercises in a state where it can do anything.

Delete that statement and the whole suite still passes. The three things that look like coverage are not:

  • The two new unit tests call the inner helper directly with hand-supplied arguments, so they prove the helper and say nothing about the wiring.
  • The one pre-existing test that now reaches this branch arrives with a GPU whose target is already populated, so the fill is a no-op there and its assertions cannot move either way.
  • The new scenario asserts entry count and bus address, but never reads the target field the fill exists to populate.

That field is user-observable output of the machine-readable report, and the contributor rules on the base branch are explicit that a unit test on an internal helper does not discharge the scenario requirement - it proves the function, not the behaviour. So the entire second half of this change could be dropped in a later refactor with every gate green.

The enumeration half is a different matter and is well built: the class-based selection is covered, the vendor gate keeps non-AMD accelerators out of the counts, the node ordering makes the topology read deterministic rather than flaky, and the count guard can only ever suppress a fill, never produce a wrong one. This objection is only about the fill being unpinned.

Concrete fix: extend the new step, or add one, to assert that every AMD entry carries a non-empty target whenever the kernel topology is uniform and its GPU-node count equals the number of AMD entries reported - no-opping on the same premise-failure conditions the step already uses. And give the fill a seam so a unit test can drive the early-return branch against a planted topology instead of the host's, the way the topology reader itself is already split.

One related observation rather than part of the objection: four checks are red at this head. I could not attribute any of them to a defect I found. The one candidate is the new scenario's strict count equality, which can fail on a legitimate host - see the non-blocking items - but that is inference and cannot be confirmed from here.

The full round of findings, including five non-blocking items, is in the review comment posted alongside this one.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read through the lspci class widening and the KFD gap-fill logic, checked out this commit locally, and ran the unit tests plus clippy against it. The core fix is solid: is_lspci_gpu_line now correctly picks up class 1200 ("Processing accelerators") devices, and uniform_kfd_gfx_target_in/fill_missing_gfx_targets only fill a missing gfx_target when every KFD node agrees and the node count matches the GPU count exactly, refusing otherwise rather than guessing. I hand-checked the GC-version decode arithmetic in the new test fixtures (90402 -> gfx942, 110003/110000 -> gfx1103/gfx1100 correctly refusing to fill) and it's right. All existing and new unit tests pass, clippy is clean.

But this PR's own new e2e scenario (examine-16) is failing on real hardware in this repo's CI, and it looks like a real problem rather than test flakiness:

(Strix Halo Ubuntu also failed, but that's the same pre-existing VRAM-preflight infra issue seen on other PRs in this queue, unrelated to this change.)

Looking at the MI300X run in detail: lspci correctly enumerates all 8 physical Instinct GPUs with real, distinct PCI addresses, so the fix itself works as intended. But the same run's JSON shows only one render device (/dev/dri/renderD152) and the KFD topology exposes only one node with a nonzero gfx_target_version. That points to these being shared bare-metal runners: the host has 8 MI300X GPUs, lspci walks the whole PCI bus and sees all of them, but this particular job is only cgroup-granted access to one via KFD/render-node restrictions.

That's more than a flaky test. Before this PR, class-1200 devices weren't matched at all, so a host like this fell through to the old KFD-based fallback, which is scoped to what's actually accessible. Now lspci's broader PCI-bus view wins, so on any host where PCI visibility is wider than device-cgroup visibility (shared bare-metal, some container/VM setups), examine will report every physically-present GPU as usable even when this process can only touch one of them. Left an inline note on the new scenario, but the fix likely needs to happen in probe_gpus_lspci or how gpus[]/has_amd_gpu are meant to be interpreted on hosts like this, not just in the test.

Reading lspci's processing-accelerator class is what finally finds Instinct
parts, but it also changed *who decides* which GPUs exist, and the PCI bus
answers a different question from the kernel. In a container the host bus
still carries every card the node has while KFD describes only the one
passed through, so `examine --json` listed eight MI300X GPUs where one was
usable. That is what examine-16 caught on four lanes -- "the kernel
describes 1 GPU node(s) but the report lists 8 AMD GPU(s)" -- and the test
was right: `rocm serve` would have failed on a GPU `examine` had just
advertised, which is worse than the under-detection this replaced, because
that at least failed honestly.

So keep the reported list as the set the kernel exposes and use the PCI
scan to enrich it. `kfd_gpu_nodes_in` reads the KFD topology per node and
returns each node's own gfx target alongside the PCI address it reports;
`apply_kernel_gpu_membership` then matches those against what lspci found,
by address rather than by position. A matched node keeps lspci's marketing
name and APU/discrete verdict and fills only an absent target. A node the
PCI scan did not list is still reported -- it is usable, so dropping it
would under-report exactly the way this whole change exists to fix -- just
without the enriched name. Devices on the bus that the kernel does not
expose leave `gpus[]` but not the report: they are named in a note, because
"the bus has it and the kernel does not" is itself a diagnosis.

Two deliberate non-changes. An empty topology is a no-op rather than "no
GPUs", so a host whose `amdgpu` never bound keeps its PCI list and the
driver probes explain why nothing is usable. And only `is_amd` entries are
touched, since KFD says nothing about an NVIDIA card.

KFD states an address as `domain` plus `location_id`, the kernel's
`pci_dev_id()` -- `(bus << 8) | devfn`. The decode is checked against
hardware, not against itself: an 8-GPU MI300X host reports location_id 4352,
12032, 17920, 23808, 35584, 43520, 49664 and 55808, which decode to exactly
the eight addresses `lspci -D` prints. A zero location_id is refused rather
than decoded to the host bridge.

This also removes the gfx-target gap-fill added in the previous commit,
which review flagged as wired in by a statement no test could exercise.
That is confirmed: deleting the call leaves the whole workspace suite green
and turns the entire apparatus into four "never used" warnings, because the
two unit tests called the inner function directly. It is not worth a test,
because reading targets per node supersedes it -- each card takes the target
of its own node, so the uniformity guard the gap-fill needed to avoid
stamping an APU's target onto a dGPU is no longer a guard but a consequence.

Covered by nine unit tests, including the eight-on-the-bus/one-in-the-kernel
case and a planted-topology test that drives the sysfs read and the
reconcile together. Verified end to end on an 8-GPU MI300X host: natively it
reports all eight with their own addresses and targets and examine-16
passes, and with the topology bind-mounted down to a single node inside a
mount namespace -- the container shape -- examine-16 fails on the previous
commit with the exact CI message and passes here, reporting one GPU at
0000:5d:00.0 with the other seven noted.

EAI-8449

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r
r0x0r requested a review from juhovainio September 25, 2026 11:35
@r0x0r

r0x0r commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator Author

Fixed in 407d8fc6 — and your read of the cause was right, including where the fix belonged.

gpus[] is now the set the kernel exposes, with lspci used only to name those entries rather than to decide membership. KFD turns out to state each node's PCI address itself (domain + location_id, where location_id is (bus << 8) | devfn), so the two sources are matched by address, not position — positional matching would mis-name GPUs on exactly the hosts where the lists diverge. A node the kernel exposes but lspci cannot name is still listed, with its real pci_id and a note; bus devices the kernel does not expose move to a note rather than disappearing.

examine-16 is unchanged. It was failing because it was correct.

Verified on hardware rather than reasoned about: the devbox is an 8-GPU MI300X host, so natively all 8 enumerate with their own addresses and gfx942. Bind-mounting the KFD topology down to a single node reproduces your cgroup-restricted shape — the pre-fix binary fails with the verbatim CI message ("the kernel describes 1 GPU node(s) but the report lists 8 AMD GPU(s)"), and the fixed binary reports one GPU at 0000:5d:00.0 with the other seven noted.

Not verified, and worth saying plainly: the real CI container (its KFD filtering may differ from my bind-mount), MI350P, and any APU+dGPU host.

I also removed the gfx-target gap-fill wiring the automated review flagged. Deleting it left the suite green and produced four function is never used warnings across the apparatus — the per-node targets supersede it.

Agreed on Strix Halo Ubuntu being the pre-existing VRAM-preflight issue; it is still failing across the queue for the same reason.

@siloteemu
siloteemu dismissed their stale review September 25, 2026 12:04

Superseded: this objection was filed against an earlier revision. The branch has been re-reviewed at its current head and a fresh change request has been filed there, so this one is withdrawn to leave a single live objection to answer.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 407d8fc

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Makes examine's AMD GPU list the set the KFD topology exposes, enriched (not decided) by the lspci scan matched on PCI address, after widening the lspci class filter to include processing accelerators. Outcome: Needs work — one reachable naming regression on the very container shape the change targets. Reviewed at check-run conclusions 20 success, 1 failure, 5 pending, 1 skipped. Verified: ran the ten new/affected unit tests in a throwaway copy (all pass) and mutated eleven individual branches of the production change one at a time in that copy — ten were killed by a named test, including three tests that die when address matching is reverted to positional, so the heart of this round is genuinely covered; the full suite, the end-to-end suite and any GPU or Windows lane were not run here. Also confirmed: the gap-fill apparatus from the middle commit is gone with no orphaned function, test or doc reference left behind; the new scenario id matches the feature file's convention and its naming lint; the expectations file has no row for it and its one adjacent exemption is unrelated and unwidened; no internal names, hosts or links in the diff; no injected instructions in code or commit messages. Blocking: 1 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

crates/rocm-core/src/examine.rs:1341-1356 (paired with :1219-1221) — on a host with no lspci but with rocminfo, every GPU now ends up named "AMD GPU (from kernel topology)" and the report carries a note saying its marketing name is unknown, even though rocminfo knows the real name. The new membership pass runs before the rocminfo probe and pushes a placeholder-named entry for each kernel node the PCI scan could not name; the rocminfo probe only adopts a marketing name when the existing entry's name is empty ("if !marketing.is_empty() && gpu.name.is_empty()"), so the placeholder wins. Before this change the list was still empty at that point and rocminfo supplied the real names. That configuration is not exotic — it is the ordinary ROCm container, which ships rocminfo but frequently not pciutils, i.e. exactly the shape this PR exists to fix; the head commit's own text cites a host where lspci is absent. The "marketing name is unknown" note is also emitted before rocminfo has had a chance to speak, so it is asserted prematurely regardless. It blocks because it is a user-visible quality regression introduced by this diff on the targeted scenario, and no test covers the ordering interaction between the two probes. Concrete fix: let rocminfo replace a kernel-topology placeholder as well as an empty name (e.g. treat the placeholder as "unnamed" in that condition, or leave the name empty in the membership pass and render a default at report time), and emit the unnamed note only once naming is final. Add a unit test that runs the membership pass with an empty PCI list and then applies a rocminfo reading, asserting the marketing name survives and no "unknown name" note is produced.

Non-blocking

  • crates/rocm-core/src/examine.rs:1400-1403 — the documented rule that an empty PCI address "matches nothing, not a wildcard" is the one branch that survived mutation: deleting the emptiness check leaves all ten tests green, so the guard is unproven (today unreachable from the lspci path, which always yields an address, but the doc states it as a rule).
  • crates/rocm-core/src/diagnose.rs:1127-1131 — the code comment says several KFD nodes may legitimately share one card's address on a partitioned Instinct; each such node clones the same PCI entry, so the AMD GPU count becomes the partition count and can trip the multi-GPU IOMMU check on a single physical card.
  • tests/e2e-cucumber/features/examine.feature:190 — the new scenario is tagged only for GPU, so it also runs on the Windows GPU lane and passes there only because a Linux-only sysfs read happens to fail; the repo's own capability code gates the equivalent read with an explicit platform cfg.
  • crates/rocm-core/src/lib.rs:5145-5155 — the node properties file is read twice per node (once for the target version, once for the address), and no test covers a node whose target comes from the standalone file while properties is absent, where the address silently becomes empty.
  • tests/e2e-cucumber/features/examine.feature — the contributor rules ask for user-facing docs to move with observable behavior; examine --json gains two new note forms and a changed gpus[] membership rule, and nothing in the README or the docs directory describes either (nothing is stale, but nothing documents the new contract either).

…cile

Review found the gfx-target work wired in by a statement no test could
exercise. Re-verified on this head: deleting `apply_kernel_gpu_membership(e,
&nodes)` left the whole workspace suite green -- 31 "test result: ok", zero
failures -- because every unit test called that inner function directly with a
hand-supplied node list, and examine-16 read only the entry count and PCI
address. Both halves of the fill were therefore unfalsifiable.

Two changes, one per half.

`probe_gpus_kernel_membership_in` gives the probe the same planted-directory
seam `kfd_gpu_nodes_in` already has, one level further out, so a test can drive
the read, the "cannot say" early return and the handoff together instead of only
the reconcile the handoff calls. The `/sys` path moves up into the probe, which
orphans `linux_kfd_gpu_nodes` -- its cfg split is now the probe's -- so it goes.

examine-16 now also asserts every AMD entry carries a non-empty `gfx_target`,
which is the observable end of the second half and what the scenario was missing
per AGENTS.md ("a unit test asserting the internal helper does NOT discharge
this; it proves the function, not the behavior"). Gated on the topology being
uniform, on the same premise-failure footing as the existing guards: where nodes
disagree, a node count cannot say which target belongs to which card, so the
step stays silent rather than assert something it has not established.

Falsified both. Deleting the handoff fails
`the_membership_probe_reconciles_the_topology_it_is_pointed_at` (left: 8, right:
1). Deleting the gap-fill on an 8-GPU gfx942 host with no `rocminfo` leaves all
eight entries with an empty `gfx_target`: the pre-existing count/PCI assertions
still pass, the new one fails.

EAI-8449

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…sing-accelerators

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…ses them

The Windows build failed with three dead-code errors. probe_gpus_kernel_membership
is cfg(any(target_os = "linux", test)), so off Linux it is the no-op stub and
KfdGpuNode, apply_kernel_gpu_membership and pci_ids_match have no reachable
caller -- but they carried no cfg of their own, so they were compiled and
immediately unused. Linux clippy cannot see this: there the real branch compiles
and every one of them is live.

Gates them with the same cfg their only callers carry, and makes the stub a
const fn, which clippy asks for once it is the definition that compiles.

Verified by flipping the feature's cfgs from linux to windows and running clippy
on Linux: three "never constructed/used" errors before, none after.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 5127ae1

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Makes examine's AMD GPU list the set the KFD topology exposes, matched by PCI address against a widened lspci class filter, and this round adds cfg gating so the KFD path is Linux-only and moves the /sys path up into the probe. Outcome: Needs work — the round-2 blocking objection is still live at this head, and one of the two new tests does not test what it names. Verified: ran the affected unit module in a throwaway copy (423 pass) and mutated seven production branches one at a time in that copy — the class widening, the membership handoff and the gfx-target gap-fill are each killed by a named test, but three mutations survived, including deletion of the early return the new unreadable-topology test is built around; the full workspace suite, the end-to-end suite and any GPU or Windows lane were not run, and the non-Linux build path was not compiled. Also confirmed: the removed topology helper leaves no orphaned reference, the renamed scenario id still satisfies the naming lint with no stale references, clippy -D warnings is clean on the library, the const fn stub is valid on the pinned toolchain, and the diff carries no internal names, hosts or links and no injected instructions. Checks at review time: 1 skipped, 20 success, no failures. Blocking: 2 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

1. crates/rocm-core/src/examine.rs:1219 (paired with :1371-1380 and :1413-1418) — the round-2 naming regression is not discharged. Nothing in this round touched the interaction. The membership pass still runs before probe_gpus_rocminfo (ordering at :395-404) and still pushes name: "AMD GPU (from kernel topology)" for every kernel node the PCI scan could not name; probe_gpus_rocminfo still adopts a marketing name only when the existing name is empty — if !marketing.is_empty() && gpu.name.is_empty() — so the placeholder wins and the real name from rocminfo is discarded. The "their marketing name is unknown" note at :1413-1418 is likewise emitted before rocminfo has spoken, so it is asserted prematurely in every case. The affected shape is the ordinary ROCm container: rocminfo present, pciutils absent — which is the shape the PR's own scope caveat describes. Before this change the list was empty at that point and rocminfo supplied real names, so this is a user-visible regression introduced by the diff on its own target scenario. It is also unpinned in either direction: widening :1219 to replace the placeholder as well as an empty name leaves all 423 tests green, so no test would notice either the bug or a fix. Concrete fix: treat the kernel-topology placeholder as unnamed in that condition (or leave the name empty in the membership pass and render a default at report time), and emit the unnamed note only after naming is final. Add a unit test that runs the membership pass against an empty PCI list, then applies a rocminfo reading, and asserts the marketing name survives and no "unknown name" note is produced.

2. crates/rocm-core/src/examine.rs:3303 — the_membership_probe_leaves_the_report_alone_when_the_topology_is_unreadable does not test the code it names. The test's own comment names the early return in probe_gpus_kernel_membership_in — "No KFD at all … is 'cannot say', not 'no GPUs'" — but replacing that early return (crates/rocm-core/src/examine.rs:1326-1328) with unwrap_or_default(), which turns "cannot read" into "read an empty topology", leaves all 423 tests green. The assertion is satisfied entirely by the separate if nodes.is_empty() { return; } guard at the top of apply_kernel_gpu_membership, a pre-existing branch the test does not name. So the distinction the test exists to protect — unreadable versus empty — is unprotected, and a future refactor that collapses the two would pass. Concrete fix: make the assertion discriminate, e.g. drive apply_kernel_gpu_membership with an empty node list and the probe with an unreadable directory in the same test and assert they are both no-ops for their own reasons, or have the unreadable case assert on something the empty-guard path cannot produce (the absence of the note text, plus an observable marker that the reconcile was never entered).

Non-blocking

  • crates/rocm-core/src/examine.rs:1426-1429 — the documented rule that an empty PCI address "matches nothing, not a wildcard" still survives mutation: deleting !left.is_empty() && leaves every test green, so the guard remains unproven (unreachable today because the left side is always the lspci-sourced address).
  • tests/e2e-cucumber/features/examine.feature:193-196 — the comment promises the step no-ops on "a topology whose nodes disagree on a target", but assert_gpus_match_kfd_nodes skips only the gfx-target loop in that case; the node-count and non-empty pci_id assertions still run, so the prose overstates the guard the code provides.
  • tests/e2e-cucumber/features/examine.feature:197 — the scenario is tagged @requires-gpu only while the repo already uses @requires-os:linux on six other scenarios; the Rust side was correctly cfg-gated this round, but the scenario still runs on a Windows GPU lane and passes there only because a Linux-only read happens to fail. Nothing in the repo shows lspci is installed on the GPU runners either, so the scenario may pass every run without exercising the new path.
  • crates/rocm-core/src/examine.rs:1083-1088 — the class filter matches anywhere in the line rather than the class field before ]:, so a device whose marketing name contains one of the four phrases would be admitted; the new test only covers a bridge line with a different class string.
  • crates/rocm-core/src/lib.rs:5148-5152 — properties is read a second time for the address after the target version was taken from either source, so a node whose target comes from the standalone file while properties is absent is counted as a GPU but silently gets an empty pci_id, and no test covers it. The new gpus[] membership rule and the two new note forms are also undocumented in the README and docs directory.

@siloteemu
siloteemu dismissed their stale review September 29, 2026 14:21

Superseded by a new round at 5127ae1. The naming objection is restated there because it is still live at this head, together with a second finding, so there is only one change request to answer.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 5127ae1

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Two blocking findings at this head. The full round, including five non-blocking points, is in the comment on this PR.

1. crates/rocm-core/src/examine.rs:1219 (with :1371-1380 and :1413-1418) — the naming regression raised last round is still live. Nothing in this round touched the interaction. The membership pass still runs before probe_gpus_rocminfo (ordering at :395-404) and still pushes name: "AMD GPU (from kernel topology)" for every kernel node the PCI scan could not name. probe_gpus_rocminfo adopts a marketing name only when the existing name is empty — if !marketing.is_empty() && gpu.name.is_empty() — so the placeholder wins and the real name from rocminfo is discarded. The "their marketing name is unknown" note at :1413-1418 is likewise emitted before rocminfo has spoken, so it is asserted prematurely in every case.

The affected shape is the ordinary ROCm container: rocminfo present, pciutils absent — the same shape this PR's own scope caveat describes. Before this change the list was empty at that point and rocminfo supplied real names, so this is a user-visible regression the diff introduces on its own target scenario. It is unpinned in either direction: widening :1219 to replace the placeholder as well as an empty name leaves all 423 tests green, so no test would notice the bug or a fix.

Fix: treat the kernel-topology placeholder as unnamed in that condition (or leave the name empty in the membership pass and render a default at report time), and emit the unnamed note only after naming is final. Add a unit test that runs the membership pass against an empty PCI list, then applies a rocminfo reading, and asserts the marketing name survives and no "unknown name" note is produced.

2. crates/rocm-core/src/examine.rs:3303 — the_membership_probe_leaves_the_report_alone_when_the_topology_is_unreadable does not test the code it names. The test's comment names the early return in probe_gpus_kernel_membership_in — "No KFD at all … is 'cannot say', not 'no GPUs'" — but replacing that early return (:1326-1328) with unwrap_or_default(), which turns "cannot read" into "read an empty topology", leaves all 423 tests green. The assertion is satisfied entirely by the separate if nodes.is_empty() { return; } guard at the top of apply_kernel_gpu_membership, a pre-existing branch the test does not name. So the distinction the test exists to protect — unreadable versus empty — is unprotected, and a future refactor collapsing the two would pass.

Fix: make the assertion discriminate — drive apply_kernel_gpu_membership with an empty node list and the probe with an unreadable directory in the same test and assert both are no-ops for their own reasons, or have the unreadable case assert on something the empty-guard path cannot produce (the absence of the note text, plus an observable marker that the reconcile was never entered).

A test that does not test the thing it names gates a merge here rather than being filed as a nit, because a weak test is not strengthened after it lands.

The positive half: the class widening, the membership handoff and the gfx-target gap-fill are each killed by a named test under single-line mutation, the cfg gating added this round is correct, the removed topology helper leaves no orphaned reference, and the library is clean under clippy -D warnings.

…sing-accelerators

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…r (EAI-8449)

The membership pass runs before probe_gpus_rocminfo and stamped
"AMD GPU (from kernel topology)" on every kernel node the PCI scan could not
name. rocminfo then adopted a marketing name only when the existing one was
empty, so the placeholder won and the real model name was discarded -- on the
ordinary ROCm container with rocminfo but no pciutils, which is the shape this
PR exists for. Before the membership pass that list was empty there and rocminfo
supplied real names, so the diff regressed its own target case.

The placeholder is now one named constant that gpu_name_is_unknown() reads back,
so a later probe treats it as the absence of a name that it is. The "marketing
name unknown" note moved after rocminfo, since it was previously asserted before
the probe most likely to refute it had spoken.

the_membership_probe_leaves_the_report_alone_when_the_topology_is_unreadable
named an early return it did not test: replacing that return with
unwrap_or_default(), collapsing "cannot say" into "KFD listed no GPU", left all
423 tests green because apply_kernel_gpu_membership's own empty-guard absorbed
it. probe_gpus_kernel_membership_in now reports which answer it got, so the test
can discriminate.

Falsified: reverting the name condition fails the new placeholder test; the
unwrap_or_default() collapse fails the unreadable test. Each mutation reddens
only its own test.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@volen-silo
volen-silo dismissed stale reviews from juhovainio and siloteemu September 30, 2026 09:39

Addressed

@volen-silo

Copy link
Copy Markdown
Collaborator

Numbering collision with #444, flagged here because neither PR will show a conflict.

Both this PR and #444 add a scenario numbered examine-16 to tests/e2e-cucumber/features/examine.feature. They add it at different points in the file, so git merges both cleanly and the duplicate only exists once the second one lands. main currently tops out at examine-15, so each PR is correct in isolation.

#444 is approved and this one is not, so on current state #444 lands first and this PR wants examine-17. That order is not guaranteed, so the rule rather than the number: whichever merges second renumbers, and the scenario ids in this repository are positional and per-feature-file, so the renamed scenario also has to sit in the right place rather than only carry the right number.

Raising it rather than changing it, since the branch is yours. Happy to push the renumber if you would rather not carry it.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bug from my earlier review is fixed, and fixed well. That review flagged: on a shared bare-metal host (an 8-GPU MI300X node passing through only one GPU to a container), lspci walks the whole PCI bus and sees every physical card, while KFD/render-node access is scoped to what's actually usable — so examine would report GPUs the process can't actually touch.

Commit 407d8fc restructures the fix correctly: instead of PCI deciding which GPUs exist and KFD only filling gaps, the kernel topology now decides membership (which respects cgroup/passthrough restrictions), with PCI just supplying the marketing name for matched devices by PCI address — not position, which also fixes a latent bug on partitioned Instinct hosts where several KFD nodes can share one physical card's address. Unexposed-but-on-bus devices are kept in a note rather than silently dropped, which is the right call ("the bus has it and the kernel doesn't" is itself a diagnosis). The two follow-up commits (Windows dead-code cfg gating, rocminfo-name-beats-placeholder precedence) are both clean, narrow fixes.

Verified: read the reconciliation logic and its tests in full, ran cargo test -p rocm-core --lib examine at this head myself (53 passed, 0 failed). Real-hardware CI confirms it in practice — the MI300X and rad3 R9700 e2e runs that failed under the old logic ("kernel describes 1 GPU node but the report lists 8 AMD GPU(s)") are now green. Approving.

@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 5d7c3cd

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Widens the lspci class filter so Instinct parts (PCI class 1200, "Processing accelerators") are enumerated, then makes the reported AMD GPU list the set the kernel's KFD topology exposes — named and enriched from the PCI scan, matched by address — and adds a pass that reports which kernel-sourced cards ended up unnamed. Outcome: Needs work — one ordering invariant that exists only in prose, introduced by the very commit that fixes the prior round's naming regression. Verified: ran the crate's library tests on a scratch copy (424 pass) and mutated fifteen individual branches of the change one at a time in that scratch copy — all mutation experiments ran on a scratch copy; the tree under review was never modified. Both blocking findings from earlier rounds are genuinely discharged: reverting the placeholder-versus-marketing-name condition is killed by a named new test, and replacing the topology read's early return with a lossy default is now killed by the test that names it, which the added result enum makes discriminating. Also killed by named tests: the class widening, address-based rather than positional matching, the never-overwrite rule on an already-resolved target, the empty-topology guard, the zero-address refusal, the unexposed-device note, and the node sort. Confirmed independently: the PCI address decode matches the three real addresses it claims; the base branch's scenario numbering stops one below the number added here and the repo's own naming lint requires sequential per-file numbering with suite-wide unique indexes; no internal names, hosts, links or registry paths anywhere in the diff; no injected instructions in code, comments or commit messages. Not checked in this round: the end-to-end suite, any GPU or Windows lane, and cross-compilation for a non-Linux target. Reviewed against 26 successful checks, 1 skipped and 1 pending, with zero failures. Blocking: 1 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

crates/rocm-core/src/examine.rs:1489-1500 (call site at :405) — the ordering contract that makes the head commit's fix work is stated only in a doc comment, and every way of breaking it leaves the whole suite green.

The head commit correctly splits the "marketing name is unknown" note out of the membership pass into note_unnamed_kernel_topology_gpus, because the membership pass runs before rocminfo and so cannot know whether a card is finally unnamed. That function's own doc says: "Must run after probe_gpus_rocminfo and before probe_gpus_sysfs_fallback: the first is what resolves the placeholder, and the second plants the same placeholder again with a note of its own, which this would otherwise count a second time." Both halves of that sentence are unenforced:

  • Moving the call back above probe_gpus_rocminfo — which reinstates exactly the premature-note defect this commit exists to remove — leaves all 424 tests passing.
  • Moving it below probe_gpus_sysfs_fallback, the double-count the comment warns about, leaves all 424 passing.
  • Deleting the call outright leaves all 424 passing.

The naming half of the same fix is properly pinned: narrowing gpu_name_is_unknown back to an emptiness check is killed by a_rocminfo_marketing_name_outranks_the_kernel_topology_placeholder. But that test invokes the membership pass, the rocminfo fold and the note function by hand, in the right order, so it proves the three functions and says nothing about the sequence the product actually runs. This is the shape the fix itself created: before this commit the note was emitted inside a pass that tests drive directly; now it is a free-floating call whose correctness is entirely positional, with the position asserted in prose only. It blocks because a user-visible note on the machine-readable report can silently regress to firing on hosts where every card is named, and because this is the third round in which a guard on this code path has been stated rather than proven.

Concrete fix: give the ordering an assertion rather than a comment. The cheapest form is a unit test that drives the three passes in the wrong order and asserts the bad outcome is impossible — but that only restates the prose. Better: make the invariant structural so it cannot be violated silently, e.g. have note_unnamed_kernel_topology_gpus take a marker that only the post-rocminfo stage can produce (the existing KfdTopology value is already threaded through and discarded at :1372), or fold the note emission into a small stage function that calls probe_gpus_rocminfo and the note together so the sequence is one unit. Note that the naive fix — simply adding a unit test that calls the three in the documented order — does not close this: that is what the existing test already does, and it is precisely what survived all three mutations. The assertion has to be able to observe the order, not assume it.

Non-blocking

  • crates/rocm-core/src/examine.rs:405 — deleting the probe_gpus_kernel_membership call from the probe chain still leaves all 424 library tests green; the new probe-level test pins the handoff one level in, but the top-level wiring rests entirely on the new scenario, which no-ops on several premises.
  • crates/rocm-core/src/examine.rs:1513 — third round running: the documented rule that an empty PCI address "matches nothing, not a wildcard" is unproven; deleting the emptiness check leaves every test green.
  • tests/e2e-cucumber/features/examine.feature:197 — still tagged only for GPU, so it also runs on the Windows GPU lane and passes vacuously there because a Linux-only sysfs read fails; the suite has a Linux platform tag used by several sibling scenarios, and the Rust side of this same concern was gated in an earlier commit while the scenario tag was not.
  • The pull request description is stale in the direction that makes the change look better covered than it is: its "What changed" section describes a helper plus a guarded gap-fill that no longer exist in the tree, its test plan names three tests that are not present, and it names a scenario identifier different from the one actually added.
  • crates/rocm-core/src/diagnose.rs:383-385 with :1126-1131 — the AMD GPU count is now the kernel node count, and the membership pass's own comment says several nodes may legitimately share one card's address on a partitioned Instinct, so one physical card can read as several GPUs and trip the multi-GPU advice that asks the user to edit the bootloader and reboot; carried from the prior round and still open.

On the other reviewer's numbering point: the rule as stated matches the repository's own naming lint, which requires per-file sequential numbering and suite-wide unique indexes, and this branch is correct in isolation — the base branch stops one number below and the new scenario is appended last. The interaction with the other pull request was not verified in this round.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pr-review-watcher · 5d7c3cd

Requesting changes on one item. The rest of this round's findings are non-blocking and are in the round comment posted alongside this.

The ordering contract that makes the head commit's fix work is stated only in a doc comment, and every way of breaking it leaves the whole suite green.

The head commit is right to split the "marketing name is unknown" note out of the kernel-membership pass: that pass runs before the rocminfo probe, so it cannot know whether a card is finally unnamed. The new function's own doc states the constraint — it must run after the rocminfo probe and before the sysfs fallback, because the first resolves the placeholder and the second plants the same placeholder again with a note of its own. Both halves of that sentence are unenforced. Measured on a scratch copy, with the tree under review left unmodified:

  • Moving the call back above the rocminfo probe — which reinstates exactly the premature-note defect this commit exists to remove — leaves all 424 library tests passing.
  • Moving it below the sysfs fallback, the double-count the comment warns about, leaves all 424 passing.
  • Deleting the call outright leaves all 424 passing.

The naming half of the same fix is properly pinned: narrowing the unknown-name predicate back to an emptiness check is killed by the new test that names it. But that test invokes the membership pass, the rocminfo fold and the note function by hand in the right order, so it proves the three functions and says nothing about the sequence the product actually runs.

Why this gates rather than waits: it is the shape the fix itself created. Before this commit the note was emitted inside a pass that tests drive directly; now it is a free-floating call whose correctness is entirely positional, with the position asserted in prose only. A user-visible note on the machine-readable report can therefore regress silently to firing on hosts where every card is named, and a weak test is not strengthened after a merge — nobody reopens a landed change to do it.

Suggested fix, with the caveat that matters most: the naive fix does not close this. Adding a unit test that calls the three functions in the documented order is what the existing test already does, and it is precisely what survived all three mutations — an assertion that assumes the order cannot observe it. Make the invariant structural instead, so it cannot be violated silently: have the note function require a value only the post-rocminfo stage can produce (the topology value is already threaded through and currently discarded), or fold the probe and the note into one small stage function so the sequence is a single unit. Either makes the three mutations above fail to compile or fail a test, rather than passing quietly.

Two smaller things in the same area are non-blocking but worth reading together with this, since they are the same pattern at lower stakes: deleting the kernel-membership call from the probe chain also leaves all 424 green, and the documented rule that an empty PCI address matches nothing rather than acting as a wildcard is unproven for the third round running.

…lback (EAI-8449)

The position `note_unnamed_kernel_topology_gpus` must hold -- after
`probe_gpus_rocminfo`, before `probe_gpus_sysfs_fallback` -- was stated
only in a doc comment. Moving the call above `rocminfo` (which reinstates
the premature-note defect the pass was split out to remove), moving it
below the fallback, and deleting it outright each left all 424 tests
passing: the existing coverage calls the passes directly in an order it
chooses itself, so it cannot see the real call site running them in
another.

Gather the passes that follow the PCI scan into `probe_gpus_after_lspci`,
so the order exists in exactly one place, and give it a `GpuProbeSources`
seam for the three things it reads -- the KFD nodes directory, a
`rocminfo` listing, and the sysfs gfx target -- so the real sequence is
drivable against a planted host. All three mutations now fail.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 3df0c04

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Makes rocm examine enumerate Instinct GPUs from lspci's processing-accelerator class, reconciles that list against the kernel's KFD topology, and — in this round — gathers the post-lspci passes into one ordered stage with an injectable source seam so the ordering the fix depends on is pinned by assertion rather than prose. Outcome: no blocking findings. Verified: on a separate scratch copy (the tree under review was not modified — git status clean at 3df0c041), cargo test -p rocm-core --lib baseline green (425 pass), then six single-branch mutations of the new sequence — note moved above the rocminfo fold, note deleted, note moved below the sysfs fallback, membership call deleted, membership swapped below rocminfo — each reddened the_unknown_name_note_is_taken_between_rocminfo_and_the_sysfs_fallback and nothing else; the sixth, inverting the empty-PCI-address rule into a wildcard, left all 425 green. Also confirmed: the decode/format claims for location_id, that no new unit test reads the real machine's devices, that every planted-topology test cleans up before asserting, and that the diff carries no internal-reference leak. Not checked this round: the Windows build, the @requires-gpu lane, and any full-workspace or e2e run. Checks at review time: 26 success, 1 skipped, 1 pending, 0 failure. Blocking: 0 · Non-blocking: 6.

Prior round

  • Blocking — "the ordering contract that makes the head commit's fix work is stated only in a doc comment, and every way of breaking it leaves the whole suite green." Discharged. Our claim was right and the remediation takes the harder of the two routes we suggested rather than the naive one we warned against: the four post-lspci passes are now a single stage function with a named source seam, so a test drives the real sequence instead of choosing its own order. All three mutations we measured now fail, and the new test is the only test that reddens for each. We also ran a fourth we had not named — swapping the membership pass below the rocminfo fold — and that fails too. The blocking concern is closed.
  • Non-blocking — "deleting the kernel-membership call from the probe chain also leaves all 424 green." Discharged, as a side effect rather than by direct aim: deleting that call now fails the same new test (scenario B loses the entry the note exists to describe), and additionally raises a dead_code warning that the repo's -D warnings clippy gate would reject.
  • Non-blocking — "the documented rule that an empty PCI address matches nothing rather than acting as a wildcard is unproven for the third round running." Still live, now the fourth. pci_ids_match's doc says "An empty address matches nothing: it means 'unknown', not 'wildcard'"; rewriting the body to left.is_empty() || left.eq_ignore_ascii_case(right) — the exact inversion the sentence forbids — leaves all 425 tests green. One correction to our own framing, in the author's favour: the guard is currently unreachable from production, because the only thing populating gpus[] before the membership pass is the lspci scan, and that takes the address from split_whitespace().next(), which cannot be empty on a line that matched. So this is a defensive invariant, not a live bug, and the honest fix is a three-line direct unit test on pci_ids_match (or deleting the guard and recording why it cannot fire) rather than a behavioural one.

🚫 Blocking (must fix before merge)

None.

Non-blocking

  • Head commit message (3df0c041), last line before the sign-off — the commit carries a generated-tooling co-author attribution footer, which the contributor guide's PR-conduct section asks contributors to avoid. It is the only commit in the range that carries one; the other eight are clean, so this reads as an accident rather than a practice. Cheap to drop on the next amend and permanent otherwise — noted rather than gated, since it has no effect on behaviour.

  • crates/rocm-core/src/examine.rs:1567 — the empty-address rule carried over from the prior round (see above); a direct unit test on pci_ids_match closes it, and this is a narrower successor to that item, not a new demand.

  • The PR description's "What changed" §2 and two of its test-plan checkboxes still describe the uniform_kfd_gfx_target_in guarded gap-fill and its two tests, which a later commit in this same PR removed outright — neither the function nor either named test exists at this head, so a reader is told the guard is the important part of a mechanism that is gone.

  • The PR description cites the new scenario's @id: as …-with-its-pci-address, but the feature file declares …-with-its-address-and-target, and the description's account of what the scenario asserts predates the gfx_target assertion a later commit added; the disclosed gap is accurate in substance but names something that is not in the tree.

  • crates/rocm-core/src/examine.rs:1077-1081 — widening the class filter to processing accelerators also admits AMD-vendor devices in that class that are not GPUs, including the on-package AI accelerator on recent AMD laptop parts (I could not confirm its exact enumeration on hardware here). On a host with a readable topology it is dropped from gpus[] but then named in the "on the bus but not exposed by the kernel … in a container this is expected" note, which would be wrong on bare metal; with no readable topology it stays in gpus[]. Narrowing the processing-accelerator branch to the GPU vendor id rather than the broad AMD match would discriminate it cleanly.

  • tests/e2e-cucumber/tests/e2e/examine_steps.rs:768-787 — the step counts GPU nodes from properties only and silently skips a node it cannot read, while the CLI's own reader falls back to a standalone gfx_target_version file; on a kernel exposing the latter the step under-counts and the equality assertion fails for a reason unrelated to this change. Mirroring the CLI's fallback, or no-op'ing rather than skipping when properties is unreadable, keeps the premise-failure behaviour the rest of the step is careful about.

@r0x0r
r0x0r enabled auto-merge September 30, 2026 15:16
@r0x0r
r0x0r added this pull request to the merge queue Sep 30, 2026
@r0x0r

r0x0r commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

The collision is real and the rule you state is the right one — but the mechanism is not, and the difference matters for how it gets caught.

Confirmed real. main tops out at examine-15. #434 adds examine-16 - The machine-readable report lists every GPU the kernel sees; #444 adds examine-16 - The scripting form names the setup runtime folder unaided. The @id: tags differ, so scenario_indexes_are_unique_across_the_suite is not what fires first — but each PR is correct in isolation and wrong together, which is your point.

Not silent, though. Both scenarios are appended at the end of the same file, after examine-15, so the two additions overlap textually rather than landing at different points:

$ git merge-tree --write-tree 3df0c041 f9e7b82b
<tree>
100644 ... 1  tests/e2e-cucumber/features/examine.feature
100644 ... 2  tests/e2e-cucumber/features/examine.feature
100644 ... 3  tests/e2e-cucumber/features/examine.feature
$ echo $?
1

The merged blob carries conflict markers at <<<<<<< 3df0c041 / >>>>>>> f9e7b82b with both examine-16 scenarios between them. Whoever merges second gets a visible conflict and has to resolve it by hand — they will not get a clean tree with a silent duplicate. That holds under squash-merge too, which is what this repo uses (allow_merge_commit=false): a squashed #444 on main still leaves #434's merge base at the pre-merge main, so it is the same three-way merge with the same conflict.

Worth correcting because "neither PR will show a conflict" invites merging both and trusting green, and the actual failure is earlier and louder than that.

Why it cannot be pre-resolved. Renumbering one of them to examine-17 now would break whichever lands first: scenario_names_are_indexed_sequentially_per_feature (tests/e2e-cucumber/tests/feature_naming.rs:137) walks scenarios_of(file).iter().enumerate() and requires {key}-{n+1:02} in declaration order, so a gap fails exactly like a duplicate. There is no numbering that is valid both before and after the other PR lands. So your rule is the only available one: whichever merges second renumbers to examine-17 and sits last in the file, since the index is positional rather than just unique.

Both are mine and both are approved as of this morning, so I will carry the renumber on whichever goes second — no need for you to push it. Thanks for catching it; a duplicate index is the kind of thing that only surfaces as a confusing feature_naming failure on an unrelated PR later.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Sep 30, 2026
@r0x0r
r0x0r added this pull request to the merge queue Oct 1, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 1, 2026
…sing-accelerators

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

# Conflicts:
#	tests/e2e-cucumber/features/examine.feature
@r0x0r

r0x0r commented Oct 1, 2026

Copy link
Copy Markdown
Collaborator Author

Carried the renumber, as promised above — #444 merged first (9d261bd8), so this PR is the one that moves.

811b58d4 merges current main in (no rebase, so the approved commits are untouched) and resolves the one conflict in tests/e2e-cucumber/features/examine.feature:

  • #444's examine-16 — The scripting form names the setup runtime folder unaided — is kept exactly as it landed on main.
  • This PR's scenario becomes examine-17 — The machine-readable report lists every GPU the kernel sees — and now sits after it, since the index is positional as well as unique. Its @id:examine-lists-every-gpu-with-its-address-and-target tag is unchanged, so nothing that references the scenario by id has to move.

As predicted in the thread above, git did not merge this silently: both scenarios append after examine-15, so the merge stopped with real conflict markers rather than producing a duplicate index.

Verified on the Linux devbox at the merge commit, not inferred:

  • cargo test --workspace --all-targets — 3200 passed, 0 failed, including scenario_names_are_indexed_sequentially_per_feature and scenario_indexes_are_unique_across_the_suite.
  • cargo clippy --workspace --all-targets -- -D warnings — clean, plus cargo clippy -p e2e-cucumber --test e2e -- -D warnings separately, since the e2e target is test = false and --all-targets does not reach it.
  • cargo fmt --all -- --check — clean.

Falsified rather than assumed: putting the index back to examine-16 turns both naming tests red with index "examine-16" is used by both examine.feature and examine.feature, so the renumber is load-bearing rather than cosmetic.

The merge also pulls in #457, #441, #450 and #415. None of them touch crates/rocm-core/src/examine.rs; the only other file this PR and #444 both changed is examine_steps.rs, which auto-merged — I checked that both sides' step definitions survived (it lists one AMD GPU per kernel GPU node… alongside #444's three) and that no step attribute is duplicated.

@r0x0r
r0x0r added this pull request to the merge queue Oct 1, 2026
Merged via the queue into main with commit 48621a8 Oct 1, 2026
29 of 30 checks passed
@r0x0r
r0x0r deleted the eai-8449-lspci-processing-accelerators branch October 1, 2026 20:32
volen-silo added a commit that referenced this pull request Oct 2, 2026
… first

HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.

The search stopped at the first match and looked for one library only, so
the second copy could not be seen at all. It is now one walker serving
both searches, recording every copy found, plus the same for the HIP
runtime so the two can be compared. Attributing each copy to the
installation that owns it is deferred; comgr_matches_runtime stays
permanently unset until that lands.

The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why the loader path the CLI sets for its own
processes walks that directory. Searching the root alone therefore missed
the copy this CLI installs itself, which is the case the entry exists for.

That knowledge now lives in one place. collect_managed_runtime_library_paths
is shared by the comgr search and the loader path, so a managed runtime's
layout is described once rather than twice. A second description is a
second thing to keep correct, and the two had already diverged.

Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.

The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record the predictable venv layout is read
directly, and a miss costs a directory that is simply not reported.

No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.

Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.

The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.

Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.

The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.

Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.

Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>

Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.
volen-silo added a commit that referenced this pull request Oct 2, 2026
… first

HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.

The search stopped at the first match and looked for one library only, so
the second copy could not be seen at all. It is now one walker serving
both searches, recording every copy found, plus the same for the HIP
runtime so the two can be compared. Attributing each copy to the
installation that owns it is deferred; comgr_matches_runtime stays
permanently unset until that lands.

The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why the loader path the CLI sets for its own
processes walks that directory. Searching the root alone therefore missed
the copy this CLI installs itself, which is the case the entry exists for.

That knowledge now lives in one place. collect_managed_runtime_library_paths
is shared by the comgr search and the loader path, so a managed runtime's
layout is described once rather than twice. A second description is a
second thing to keep correct, and the two had already diverged.

Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.

The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record the predictable venv layout is read
directly, and a miss costs a directory that is simply not reported.

No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.

Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.

The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.

Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.

The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.

Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.

Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>

Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.
volen-silo added a commit that referenced this pull request Oct 2, 2026
… first

HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.

The search stopped at the first match, so the second copy could not be
seen at all. It now records every copy found, in loader search order.
Attributing each copy to the installation that owns it, and doing the
same search for the HIP runtime so the two can be compared, is deferred;
comgr_matches_runtime stays permanently unset until that lands.

The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why `managed_sdk_ld_library_path` (the
environment the CLI builds to run an SDK tool, such as
rocm_agent_enumerator, inside a managed runtime) walks that directory.
Searching the root alone therefore missed the copy this CLI installs
itself, which is the case the entry exists for.

That knowledge now lives in one place for this search:
collect_managed_runtime_library_paths is shared by the comgr search here
and by managed_sdk_ld_library_path, so that one layout is described once
there rather than twice. A third description, collect_runtime_environment_paths,
still builds the environment a served engine process gets; unifying all
three is follow-up work, not part of this search.

Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.

The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record, root's own `_rocm_sdk_*` name is read
to find its parent -- the site-packages directory its siblings share --
and a miss costs a directory that is simply not reported.

No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.

Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.

The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.

Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.

The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.

Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.

Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.

Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.

Review: four more issues. (1) comgr_selected and the multi-copy note named
the ld.so.cache system copy as "would load" whenever a managed wheel
runtime was active but this process's own LD_LIBRARY_PATH was empty --
wrong, because rocm serve/rocm chat prepend the active runtime's own
library directories onto LD_LIBRARY_PATH before launching the engine, so
the wheel copy is what a served process actually loads. probe_comgr now
takes the active runtime's interpreter (the same FrameworkInterpreter
already threaded through probe_framework) and searches its library_paths
first, ahead of the ambient environment; pinned by
the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The
site_packages: None fallback in collect_managed_runtime_library_paths
guessed a venv layout (root/lib/<python>/site-packages) that the installer
never produces -- root is one of the runtime's own _rocm_sdk_* package
directories, not a venv root, so the guess always found nothing. It now
reads root.parent() instead, which is the site-packages directory the
guessed layout could never reach. The test fixture
(wheel_runtime_on_disk) was built from the same wrong venv shape and so
could not have caught this; rebuilt it from the real installer layout
(ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed
a_wheel_runtime_with_no_registry_record_is_still_found fails against the
old fallback before passing against the fix. (3) probe_comgr,
comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so
the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the
selection, and the multi-copy note could all break with the rest of the
suite still green. Split the ordering/selection logic into a pure
find_comgr_copies, the ldconfig -p line parsing into
parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings
(opt directory taken as a parameter rather than hardcoded, so a test can
point it at a temp folder instead of the real /opt on the machine running
the suite); each is unit-tested directly, and each new test was confirmed
to fail against a deliberately broken version of the function it covers
before being restored. (4) This message previously claimed a HIP copy
list and a single shared description of the loader path launched
processes use; neither matches this diff. Corrected above.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
volen-silo added a commit that referenced this pull request Oct 2, 2026
… first

HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.

The search stopped at the first match, so the second copy could not be
seen at all. It now records every copy found, in loader search order.
Attributing each copy to the installation that owns it, and doing the
same search for the HIP runtime so the two can be compared, is deferred;
comgr_matches_runtime stays permanently unset until that lands.

The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why `managed_sdk_ld_library_path` (the
environment the CLI builds to run an SDK tool, such as
rocm_agent_enumerator, inside a managed runtime) walks that directory.
Searching the root alone therefore missed the copy this CLI installs
itself, which is the case the entry exists for.

That knowledge now lives in one place for this search:
collect_managed_runtime_library_paths is shared by the comgr search here
and by managed_sdk_ld_library_path, so that one layout is described once
there rather than twice. A third description, collect_runtime_environment_paths,
still builds the environment a served engine process gets; unifying all
three is follow-up work, not part of this search.

Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.

The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record, root's own `_rocm_sdk_*` name is read
to find its parent -- the site-packages directory its siblings share --
and a miss costs a directory that is simply not reported.

No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.

Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.

The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.

Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.

The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.

Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.

Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.

Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.

Review: four more issues. (1) comgr_selected and the multi-copy note named
the ld.so.cache system copy as "would load" whenever a managed wheel
runtime was active but this process's own LD_LIBRARY_PATH was empty --
wrong, because rocm serve/rocm chat prepend the active runtime's own
library directories onto LD_LIBRARY_PATH before launching the engine, so
the wheel copy is what a served process actually loads. probe_comgr now
takes the active runtime's interpreter (the same FrameworkInterpreter
already threaded through probe_framework) and searches its library_paths
first, ahead of the ambient environment; pinned by
the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The
site_packages: None fallback in collect_managed_runtime_library_paths
guessed a venv layout (root/lib/<python>/site-packages) that the installer
never produces -- root is one of the runtime's own _rocm_sdk_* package
directories, not a venv root, so the guess always found nothing. It now
reads root.parent() instead, which is the site-packages directory the
guessed layout could never reach. The test fixture
(wheel_runtime_on_disk) was built from the same wrong venv shape and so
could not have caught this; rebuilt it from the real installer layout
(ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed
a_wheel_runtime_with_no_registry_record_is_still_found fails against the
old fallback before passing against the fix. (3) probe_comgr,
comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so
the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the
selection, and the multi-copy note could all break with the rest of the
suite still green. Split the ordering/selection logic into a pure
find_comgr_copies, the ldconfig -p line parsing into
parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings
(opt directory taken as a parameter rather than hardcoded, so a test can
point it at a temp folder instead of the real /opt on the machine running
the suite); each is unit-tested directly, and each new test was confirmed
to fail against a deliberately broken version of the function it covers
before being restored. (4) This message previously claimed a HIP copy
list and a single shared description of the loader path launched
processes use; neither matches this diff. Corrected above.

Review: a second round, one automated and one human, reaching the same
place by different routes.

The human reviewer confirmed, independently, that the site_packages:
None fallback added in the prior round could not actually fire:
ROCM_SDK_PROBE_SCRIPT has recorded site_packages unconditionally, outside
its try, since the probe's initial version, so every manifest that
parses at all carries Some there. The test built to cover it could not
have caught that either way, since it called the function directly with
None rather than through a real candidate. Rather than keep a reachable-
looking branch and a comment describing behaviour the code does not
have, the None arm is now a no-op (root's own directories, already
collected unconditionally above, are what a hand-constructed None caller
still gets) and the test is removed. The neighbouring root-format test
is kept and re-targeted: it already called the same function directly
with None, so it is what actually exercises that arm's contract, not the
removed test.

The automated reviewer caught a real regression the first round
introduced: three new unit tests feed a path into the `:`-split
LD_LIBRARY_PATH parser without a unix cfg gate, same as the pattern the
file already uses for that reason, and that broke the required
windows-build-and-test lane. Gated, along with the helper they share
(which would otherwise be unused, and clippy-denied, once its only
callers are gated).

It also found that the active-runtime-priority fix regresses
examine-finds-the-managed-runtimes-own-compilation-library on a healthy
host: once a managed runtime is active, its directory is found first and
labelled active-runtime, and the dedup this search has always done on
resolved path then drops the later managed-runtime hit for the identical
file -- the scenario's step asserted on the label that fix makes
unreachable. The step now accepts either label as proof the CLI's own
installed copy was found, and a new unit test
(an_active_runtime_directory_that_is_also_a_managed_root_keeps_one_label)
pins that an overlapping directory keeps exactly one label rather than
being reported twice; confirmed to fail when the active-runtime search is
moved after the others, then restored.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
volen-silo added a commit that referenced this pull request Oct 5, 2026
… first

HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.

The search stopped at the first match, so the second copy could not be
seen at all. It now records every copy found, in loader search order.
Attributing each copy to the installation that owns it, and doing the
same search for the HIP runtime so the two can be compared, is deferred;
comgr_matches_runtime stays permanently unset until that lands.

The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why `managed_sdk_ld_library_path` (the
environment the CLI builds to run an SDK tool, such as
rocm_agent_enumerator, inside a managed runtime) walks that directory.
Searching the root alone therefore missed the copy this CLI installs
itself, which is the case the entry exists for.

That knowledge now lives in one place for this search:
collect_managed_runtime_library_paths is shared by the comgr search here
and by managed_sdk_ld_library_path, so that one layout is described once
there rather than twice. A third description, collect_runtime_environment_paths,
still builds the environment a served engine process gets; unifying all
three is follow-up work, not part of this search.

Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.

The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record, root's own `_rocm_sdk_*` name is read
to find its parent -- the site-packages directory its siblings share --
and a miss costs a directory that is simply not reported.

No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.

Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.

The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.

Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.

The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.

Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.

Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.

Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.

Review: four more issues. (1) comgr_selected and the multi-copy note named
the ld.so.cache system copy as "would load" whenever a managed wheel
runtime was active but this process's own LD_LIBRARY_PATH was empty --
wrong, because rocm serve/rocm chat prepend the active runtime's own
library directories onto LD_LIBRARY_PATH before launching the engine, so
the wheel copy is what a served process actually loads. probe_comgr now
takes the active runtime's interpreter (the same FrameworkInterpreter
already threaded through probe_framework) and searches its library_paths
first, ahead of the ambient environment; pinned by
the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The
site_packages: None fallback in collect_managed_runtime_library_paths
guessed a venv layout (root/lib/<python>/site-packages) that the installer
never produces -- root is one of the runtime's own _rocm_sdk_* package
directories, not a venv root, so the guess always found nothing. It now
reads root.parent() instead, which is the site-packages directory the
guessed layout could never reach. The test fixture
(wheel_runtime_on_disk) was built from the same wrong venv shape and so
could not have caught this; rebuilt it from the real installer layout
(ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed
a_wheel_runtime_with_no_registry_record_is_still_found fails against the
old fallback before passing against the fix. (3) probe_comgr,
comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so
the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the
selection, and the multi-copy note could all break with the rest of the
suite still green. Split the ordering/selection logic into a pure
find_comgr_copies, the ldconfig -p line parsing into
parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings
(opt directory taken as a parameter rather than hardcoded, so a test can
point it at a temp folder instead of the real /opt on the machine running
the suite); each is unit-tested directly, and each new test was confirmed
to fail against a deliberately broken version of the function it covers
before being restored. (4) This message previously claimed a HIP copy
list and a single shared description of the loader path launched
processes use; neither matches this diff. Corrected above.

Review: a second round, one automated and one human, reaching the same
place by different routes.

The human reviewer confirmed, independently, that the site_packages:
None fallback added in the prior round could not actually fire:
ROCM_SDK_PROBE_SCRIPT has recorded site_packages unconditionally, outside
its try, since the probe's initial version, so every manifest that
parses at all carries Some there. The test built to cover it could not
have caught that either way, since it called the function directly with
None rather than through a real candidate. Rather than keep a reachable-
looking branch and a comment describing behaviour the code does not
have, the None arm is now a no-op (root's own directories, already
collected unconditionally above, are what a hand-constructed None caller
still gets) and the test is removed. The neighbouring root-format test
is kept and re-targeted: it already called the same function directly
with None, so it is what actually exercises that arm's contract, not the
removed test.

The automated reviewer caught a real regression the first round
introduced: three new unit tests feed a path into the `:`-split
LD_LIBRARY_PATH parser without a unix cfg gate, same as the pattern the
file already uses for that reason, and that broke the required
windows-build-and-test lane. Gated, along with the helper they share
(which would otherwise be unused, and clippy-denied, once its only
callers are gated).

It also found that the active-runtime-priority fix regresses
examine-finds-the-managed-runtimes-own-compilation-library on a healthy
host: once a managed runtime is active, its directory is found first and
labelled active-runtime, and the dedup this search has always done on
resolved path then drops the later managed-runtime hit for the identical
file -- the scenario's step asserted on the label that fix makes
unreachable. The step now accepts either label as proof the CLI's own
installed copy was found, and a new unit test
(an_active_runtime_directory_that_is_also_a_managed_root_keeps_one_label)
pins that an overlapping directory keeps exactly one label rather than
being reported twice; confirmed to fail when the active-runtime search is
moved after the others, then restored.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
volen-silo added a commit that referenced this pull request Oct 5, 2026
… first

HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.

The search stopped at the first match, so the second copy could not be
seen at all. It now records every copy found, in loader search order.
Attributing each copy to the installation that owns it, and doing the
same search for the HIP runtime so the two can be compared, is deferred;
comgr_matches_runtime stays permanently unset until that lands.

The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why `managed_sdk_ld_library_path` (the
environment the CLI builds to run an SDK tool, such as
rocm_agent_enumerator, inside a managed runtime) walks that directory.
Searching the root alone therefore missed the copy this CLI installs
itself, which is the case the entry exists for.

That knowledge now lives in one place for this search:
collect_managed_runtime_library_paths is shared by the comgr search here
and by managed_sdk_ld_library_path, so that one layout is described once
there rather than twice. A third description, collect_runtime_environment_paths,
still builds the environment a served engine process gets; unifying all
three is follow-up work, not part of this search.

Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.

The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record, root's own `_rocm_sdk_*` name is read
to find its parent -- the site-packages directory its siblings share --
and a miss costs a directory that is simply not reported.

No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.

Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.

The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.

Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.

The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.

Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.

Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.

Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.

Review: four more issues. (1) comgr_selected and the multi-copy note named
the ld.so.cache system copy as "would load" whenever a managed wheel
runtime was active but this process's own LD_LIBRARY_PATH was empty --
wrong, because rocm serve/rocm chat prepend the active runtime's own
library directories onto LD_LIBRARY_PATH before launching the engine, so
the wheel copy is what a served process actually loads. probe_comgr now
takes the active runtime's interpreter (the same FrameworkInterpreter
already threaded through probe_framework) and searches its library_paths
first, ahead of the ambient environment; pinned by
the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The
site_packages: None fallback in collect_managed_runtime_library_paths
guessed a venv layout (root/lib/<python>/site-packages) that the installer
never produces -- root is one of the runtime's own _rocm_sdk_* package
directories, not a venv root, so the guess always found nothing. It now
reads root.parent() instead, which is the site-packages directory the
guessed layout could never reach. The test fixture
(wheel_runtime_on_disk) was built from the same wrong venv shape and so
could not have caught this; rebuilt it from the real installer layout
(ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed
a_wheel_runtime_with_no_registry_record_is_still_found fails against the
old fallback before passing against the fix. (3) probe_comgr,
comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so
the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the
selection, and the multi-copy note could all break with the rest of the
suite still green. Split the ordering/selection logic into a pure
find_comgr_copies, the ldconfig -p line parsing into
parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings
(opt directory taken as a parameter rather than hardcoded, so a test can
point it at a temp folder instead of the real /opt on the machine running
the suite); each is unit-tested directly, and each new test was confirmed
to fail against a deliberately broken version of the function it covers
before being restored. (4) This message previously claimed a HIP copy
list and a single shared description of the loader path launched
processes use; neither matches this diff. Corrected above.

Review: a second round, one automated and one human, reaching the same
place by different routes.

The human reviewer confirmed, independently, that the site_packages:
None fallback added in the prior round could not actually fire:
ROCM_SDK_PROBE_SCRIPT has recorded site_packages unconditionally, outside
its try, since the probe's initial version, so every manifest that
parses at all carries Some there. The test built to cover it could not
have caught that either way, since it called the function directly with
None rather than through a real candidate. Rather than keep a reachable-
looking branch and a comment describing behaviour the code does not
have, the None arm is now a no-op (root's own directories, already
collected unconditionally above, are what a hand-constructed None caller
still gets) and the test is removed. The neighbouring root-format test
is kept and re-targeted: it already called the same function directly
with None, so it is what actually exercises that arm's contract, not the
removed test.

The automated reviewer caught a real regression the first round
introduced: three new unit tests feed a path into the `:`-split
LD_LIBRARY_PATH parser without a unix cfg gate, same as the pattern the
file already uses for that reason, and that broke the required
windows-build-and-test lane. Gated, along with the helper they share
(which would otherwise be unused, and clippy-denied, once its only
callers are gated).

It also found that the active-runtime-priority fix regresses
examine-finds-the-managed-runtimes-own-compilation-library on a healthy
host: once a managed runtime is active, its directory is found first and
labelled active-runtime, and the dedup this search has always done on
resolved path then drops the later managed-runtime hit for the identical
file -- the scenario's step asserted on the label that fix makes
unreachable. The step now accepts either label as proof the CLI's own
installed copy was found, and a new unit test
(an_active_runtime_directory_that_is_also_a_managed_root_keeps_one_label)
pins that an overlapping directory keeps exactly one label rather than
being reported twice; confirmed to fail when the active-runtime search is
moved after the others, then restored.

Review: a third round, three issues.

(1) comgr_matches_runtime was declared, defaulted, and listed in the
expected-keys test, but nothing in this diff computes it -- comparing
comgr's copies against the HIP runtime's is still deferred, same as the
commit message already said. A reader of the JSON contract alone had no
way to know the field was permanently unset rather than merely not yet
exercised by a test. Dropped from this commit entirely (the struct field,
its Default entry, its expected-keys assertion); `serde(default)` makes
re-adding it free once something actually computes it.

(2) comgr_selected named the first entry in comgr_paths as "would load"
regardless of which tier found it, so a machine carrying only a leftover
/opt/rocm-* install or an inactive managed runtime -- never a copy on
active-runtime, ld-library-path, or loader-cache -- was told that stale
copy would load, when the loader does not search /opt or a managed
runtime's directory on its own. find_comgr_copies now selects only from
those three loader-consulted tiers (LOADER_PATH_SOURCES), leaving
comgr_selected None and rewording the note when every hit is
rocm-install/managed-runtime-only; pinned by
a_search_dir_only_hit_is_not_reported_as_the_one_that_would_load. The e2e
step (assert_comgr_selection_is_stated) now asserts the same tier rule
against the real JSON: selected absent plus only non-loader-tier copies
is the pass case, not a shape to tolerate.

(3) comgr_search_dirs and managed_runtime_roots had no unit tests of
their own, and swapping the loader-cache and search-dir loops inside
find_comgr_copies left every then-existing test green -- nothing pinned
their relative order. Split comgr_search_dirs into a pure
comgr_search_dirs_in (rocm_path, the /opt directory, and the resolved
managed runtimes all taken as parameters) and added
comgr_search_dirs_labels_each_tier, covering all three source labels
against temp directories. Added loader_cache_outranks_search_dirs, built
so an ld-library-path hit cannot mask the order under test. Confirmed the
claim directly: swapped the two loops, reran the suite -- exactly
loader_cache_outranks_search_dirs failed, the other 65 tests (including
ld_library_path_outranks_loader_cache_and_search_dirs) stayed green --
then reverted the swap and reran clean before committing.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
volen-silo added a commit that referenced this pull request Oct 5, 2026
… first

HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.

The search stopped at the first match, so the second copy could not be
seen at all. It now records every copy found, in loader search order.
Attributing each copy to the installation that owns it, and doing the
same search for the HIP runtime so the two can be compared, is deferred;
comgr_matches_runtime stays permanently unset until that lands.

The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why `managed_sdk_ld_library_path` (the
environment the CLI builds to run an SDK tool, such as
rocm_agent_enumerator, inside a managed runtime) walks that directory.
Searching the root alone therefore missed the copy this CLI installs
itself, which is the case the entry exists for.

That knowledge now lives in one place for this search:
collect_managed_runtime_library_paths is shared by the comgr search here
and by managed_sdk_ld_library_path, so that one layout is described once
there rather than twice. A third description, collect_runtime_environment_paths,
still builds the environment a served engine process gets; unifying all
three is follow-up work, not part of this search.

Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.

The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record, root's own `_rocm_sdk_*` name is read
to find its parent -- the site-packages directory its siblings share --
and a miss costs a directory that is simply not reported.

No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.

Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.

The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.

Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.

The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.

Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.

Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.

Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.

Review: four more issues. (1) comgr_selected and the multi-copy note named
the ld.so.cache system copy as "would load" whenever a managed wheel
runtime was active but this process's own LD_LIBRARY_PATH was empty --
wrong, because rocm serve/rocm chat prepend the active runtime's own
library directories onto LD_LIBRARY_PATH before launching the engine, so
the wheel copy is what a served process actually loads. probe_comgr now
takes the active runtime's interpreter (the same FrameworkInterpreter
already threaded through probe_framework) and searches its library_paths
first, ahead of the ambient environment; pinned by
the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The
site_packages: None fallback in collect_managed_runtime_library_paths
guessed a venv layout (root/lib/<python>/site-packages) that the installer
never produces -- root is one of the runtime's own _rocm_sdk_* package
directories, not a venv root, so the guess always found nothing. It now
reads root.parent() instead, which is the site-packages directory the
guessed layout could never reach. The test fixture
(wheel_runtime_on_disk) was built from the same wrong venv shape and so
could not have caught this; rebuilt it from the real installer layout
(ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed
a_wheel_runtime_with_no_registry_record_is_still_found fails against the
old fallback before passing against the fix. (3) probe_comgr,
comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so
the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the
selection, and the multi-copy note could all break with the rest of the
suite still green. Split the ordering/selection logic into a pure
find_comgr_copies, the ldconfig -p line parsing into
parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings
(opt directory taken as a parameter rather than hardcoded, so a test can
point it at a temp folder instead of the real /opt on the machine running
the suite); each is unit-tested directly, and each new test was confirmed
to fail against a deliberately broken version of the function it covers
before being restored. (4) This message previously claimed a HIP copy
list and a single shared description of the loader path launched
processes use; neither matches this diff. Corrected above.

Review: a second round, one automated and one human, reaching the same
place by different routes.

The human reviewer confirmed, independently, that the site_packages:
None fallback added in the prior round could not actually fire:
ROCM_SDK_PROBE_SCRIPT has recorded site_packages unconditionally, outside
its try, since the probe's initial version, so every manifest that
parses at all carries Some there. The test built to cover it could not
have caught that either way, since it called the function directly with
None rather than through a real candidate. Rather than keep a reachable-
looking branch and a comment describing behaviour the code does not
have, the None arm is now a no-op (root's own directories, already
collected unconditionally above, are what a hand-constructed None caller
still gets) and the test is removed. The neighbouring root-format test
is kept and re-targeted: it already called the same function directly
with None, so it is what actually exercises that arm's contract, not the
removed test.

The automated reviewer caught a real regression the first round
introduced: three new unit tests feed a path into the `:`-split
LD_LIBRARY_PATH parser without a unix cfg gate, same as the pattern the
file already uses for that reason, and that broke the required
windows-build-and-test lane. Gated, along with the helper they share
(which would otherwise be unused, and clippy-denied, once its only
callers are gated).

It also found that the active-runtime-priority fix regresses
examine-finds-the-managed-runtimes-own-compilation-library on a healthy
host: once a managed runtime is active, its directory is found first and
labelled active-runtime, and the dedup this search has always done on
resolved path then drops the later managed-runtime hit for the identical
file -- the scenario's step asserted on the label that fix makes
unreachable. The step now accepts either label as proof the CLI's own
installed copy was found, and a new unit test
(an_active_runtime_directory_that_is_also_a_managed_root_keeps_one_label)
pins that an overlapping directory keeps exactly one label rather than
being reported twice; confirmed to fail when the active-runtime search is
moved after the others, then restored.

Review: a third round, three issues.

(1) comgr_matches_runtime was declared, defaulted, and listed in the
expected-keys test, but nothing in this diff computes it -- comparing
comgr's copies against the HIP runtime's is still deferred, same as the
commit message already said. A reader of the JSON contract alone had no
way to know the field was permanently unset rather than merely not yet
exercised by a test. Dropped from this commit entirely (the struct field,
its Default entry, its expected-keys assertion); `serde(default)` makes
re-adding it free once something actually computes it.

(2) comgr_selected named the first entry in comgr_paths as "would load"
regardless of which tier found it, so a machine carrying only a leftover
/opt/rocm-* install or an inactive managed runtime -- never a copy on
active-runtime, ld-library-path, or loader-cache -- was told that stale
copy would load, when the loader does not search /opt or a managed
runtime's directory on its own. find_comgr_copies now selects only from
those three loader-consulted tiers (LOADER_PATH_SOURCES), leaving
comgr_selected None and rewording the note when every hit is
rocm-install/managed-runtime-only; pinned by
a_search_dir_only_hit_is_not_reported_as_the_one_that_would_load. The e2e
step (assert_comgr_selection_is_stated) now asserts the same tier rule
against the real JSON: selected absent plus only non-loader-tier copies
is the pass case, not a shape to tolerate.

(3) comgr_search_dirs and managed_runtime_roots had no unit tests of
their own, and swapping the loader-cache and search-dir loops inside
find_comgr_copies left every then-existing test green -- nothing pinned
their relative order. Split comgr_search_dirs into a pure
comgr_search_dirs_in (rocm_path, the /opt directory, and the resolved
managed runtimes all taken as parameters) and added
comgr_search_dirs_labels_each_tier, covering all three source labels
against temp directories. Added loader_cache_outranks_search_dirs, built
so an ld-library-path hit cannot mask the order under test. Confirmed the
claim directly: swapped the two loops, reran the suite -- exactly
loader_cache_outranks_search_dirs failed, the other 65 tests (including
ld_library_path_outranks_loader_cache_and_search_dirs) stayed green --
then reverted the swap and reran clean before committing.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
volen-silo added a commit that referenced this pull request Oct 5, 2026
… first

HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.

The search stopped at the first match, so the second copy could not be
seen at all. It now records every copy found, in loader search order.
Attributing each copy to the installation that owns it, and doing the
same search for the HIP runtime so the two can be compared, is deferred;
comgr_matches_runtime stays permanently unset until that lands.

The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why `managed_sdk_ld_library_path` (the
environment the CLI builds to run an SDK tool, such as
rocm_agent_enumerator, inside a managed runtime) walks that directory.
Searching the root alone therefore missed the copy this CLI installs
itself, which is the case the entry exists for.

That knowledge now lives in one place for this search:
collect_managed_runtime_library_paths is shared by the comgr search here
and by managed_sdk_ld_library_path, so that one layout is described once
there rather than twice. A third description, collect_runtime_environment_paths,
still builds the environment a served engine process gets; unifying all
three is follow-up work, not part of this search.

Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.

The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record, root's own `_rocm_sdk_*` name is read
to find its parent -- the site-packages directory its siblings share --
and a miss costs a directory that is simply not reported.

No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.

Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.

The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.

Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.

The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.

Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.

Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.

Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.

Review: four more issues. (1) comgr_selected and the multi-copy note named
the ld.so.cache system copy as "would load" whenever a managed wheel
runtime was active but this process's own LD_LIBRARY_PATH was empty --
wrong, because rocm serve/rocm chat prepend the active runtime's own
library directories onto LD_LIBRARY_PATH before launching the engine, so
the wheel copy is what a served process actually loads. probe_comgr now
takes the active runtime's interpreter (the same FrameworkInterpreter
already threaded through probe_framework) and searches its library_paths
first, ahead of the ambient environment; pinned by
the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The
site_packages: None fallback in collect_managed_runtime_library_paths
guessed a venv layout (root/lib/<python>/site-packages) that the installer
never produces -- root is one of the runtime's own _rocm_sdk_* package
directories, not a venv root, so the guess always found nothing. It now
reads root.parent() instead, which is the site-packages directory the
guessed layout could never reach. The test fixture
(wheel_runtime_on_disk) was built from the same wrong venv shape and so
could not have caught this; rebuilt it from the real installer layout
(ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed
a_wheel_runtime_with_no_registry_record_is_still_found fails against the
old fallback before passing against the fix. (3) probe_comgr,
comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so
the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the
selection, and the multi-copy note could all break with the rest of the
suite still green. Split the ordering/selection logic into a pure
find_comgr_copies, the ldconfig -p line parsing into
parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings
(opt directory taken as a parameter rather than hardcoded, so a test can
point it at a temp folder instead of the real /opt on the machine running
the suite); each is unit-tested directly, and each new test was confirmed
to fail against a deliberately broken version of the function it covers
before being restored. (4) This message previously claimed a HIP copy
list and a single shared description of the loader path launched
processes use; neither matches this diff. Corrected above.

Review: a second round, one automated and one human, reaching the same
place by different routes.

The human reviewer confirmed, independently, that the site_packages:
None fallback added in the prior round could not actually fire:
ROCM_SDK_PROBE_SCRIPT has recorded site_packages unconditionally, outside
its try, since the probe's initial version, so every manifest that
parses at all carries Some there. The test built to cover it could not
have caught that either way, since it called the function directly with
None rather than through a real candidate. Rather than keep a reachable-
looking branch and a comment describing behaviour the code does not
have, the None arm is now a no-op (root's own directories, already
collected unconditionally above, are what a hand-constructed None caller
still gets) and the test is removed. The neighbouring root-format test
is kept and re-targeted: it already called the same function directly
with None, so it is what actually exercises that arm's contract, not the
removed test.

The automated reviewer caught a real regression the first round
introduced: three new unit tests feed a path into the `:`-split
LD_LIBRARY_PATH parser without a unix cfg gate, same as the pattern the
file already uses for that reason, and that broke the required
windows-build-and-test lane. Gated, along with the helper they share
(which would otherwise be unused, and clippy-denied, once its only
callers are gated).

It also found that the active-runtime-priority fix regresses
examine-finds-the-managed-runtimes-own-compilation-library on a healthy
host: once a managed runtime is active, its directory is found first and
labelled active-runtime, and the dedup this search has always done on
resolved path then drops the later managed-runtime hit for the identical
file -- the scenario's step asserted on the label that fix makes
unreachable. The step now accepts either label as proof the CLI's own
installed copy was found, and a new unit test
(an_active_runtime_directory_that_is_also_a_managed_root_keeps_one_label)
pins that an overlapping directory keeps exactly one label rather than
being reported twice; confirmed to fail when the active-runtime search is
moved after the others, then restored.

Review: a third round, three issues.

(1) comgr_matches_runtime was declared, defaulted, and listed in the
expected-keys test, but nothing in this diff computes it -- comparing
comgr's copies against the HIP runtime's is still deferred, same as the
commit message already said. A reader of the JSON contract alone had no
way to know the field was permanently unset rather than merely not yet
exercised by a test. Dropped from this commit entirely (the struct field,
its Default entry, its expected-keys assertion); `serde(default)` makes
re-adding it free once something actually computes it.

(2) comgr_selected named the first entry in comgr_paths as "would load"
regardless of which tier found it, so a machine carrying only a leftover
/opt/rocm-* install or an inactive managed runtime -- never a copy on
active-runtime, ld-library-path, or loader-cache -- was told that stale
copy would load, when the loader does not search /opt or a managed
runtime's directory on its own. find_comgr_copies now selects only from
those three loader-consulted tiers (LOADER_PATH_SOURCES), leaving
comgr_selected None and rewording the note when every hit is
rocm-install/managed-runtime-only; pinned by
a_search_dir_only_hit_is_not_reported_as_the_one_that_would_load. The e2e
step (assert_comgr_selection_is_stated) now asserts the same tier rule
against the real JSON: selected absent plus only non-loader-tier copies
is the pass case, not a shape to tolerate.

(3) comgr_search_dirs and managed_runtime_roots had no unit tests of
their own, and swapping the loader-cache and search-dir loops inside
find_comgr_copies left every then-existing test green -- nothing pinned
their relative order. Split comgr_search_dirs into a pure
comgr_search_dirs_in (rocm_path, the /opt directory, and the resolved
managed runtimes all taken as parameters) and added
comgr_search_dirs_labels_each_tier, covering all three source labels
against temp directories. Added loader_cache_outranks_search_dirs, built
so an ld-library-path hit cannot mask the order under test. Confirmed the
claim directly: swapped the two loops, reran the suite -- exactly
loader_cache_outranks_search_dirs failed, the other 65 tests (including
ld_library_path_outranks_loader_cache_and_search_dirs) stayed green --
then reverted the swap and reran clean before committing.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
volen-silo added a commit that referenced this pull request Oct 5, 2026
… first

HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.

The search stopped at the first match, so the second copy could not be
seen at all. It now records every copy found, in loader search order.
Attributing each copy to the installation that owns it, and doing the
same search for the HIP runtime so the two can be compared, is deferred;
comgr_matches_runtime stays permanently unset until that lands.

The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why `managed_sdk_ld_library_path` (the
environment the CLI builds to run an SDK tool, such as
rocm_agent_enumerator, inside a managed runtime) walks that directory.
Searching the root alone therefore missed the copy this CLI installs
itself, which is the case the entry exists for.

That knowledge now lives in one place for this search:
collect_managed_runtime_library_paths is shared by the comgr search here
and by managed_sdk_ld_library_path, so that one layout is described once
there rather than twice. A third description, collect_runtime_environment_paths,
still builds the environment a served engine process gets; unifying all
three is follow-up work, not part of this search.

Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.

The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record, root's own `_rocm_sdk_*` name is read
to find its parent -- the site-packages directory its siblings share --
and a miss costs a directory that is simply not reported.

No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.

Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.

The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.

Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.

The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.

Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.

Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, #444), then examine-17 (EAI-8449,
#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.

Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.

Review: four more issues. (1) comgr_selected and the multi-copy note named
the ld.so.cache system copy as "would load" whenever a managed wheel
runtime was active but this process's own LD_LIBRARY_PATH was empty --
wrong, because rocm serve/rocm chat prepend the active runtime's own
library directories onto LD_LIBRARY_PATH before launching the engine, so
the wheel copy is what a served process actually loads. probe_comgr now
takes the active runtime's interpreter (the same FrameworkInterpreter
already threaded through probe_framework) and searches its library_paths
first, ahead of the ambient environment; pinned by
the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The
site_packages: None fallback in collect_managed_runtime_library_paths
guessed a venv layout (root/lib/<python>/site-packages) that the installer
never produces -- root is one of the runtime's own _rocm_sdk_* package
directories, not a venv root, so the guess always found nothing. It now
reads root.parent() instead, which is the site-packages directory the
guessed layout could never reach. The test fixture
(wheel_runtime_on_disk) was built from the same wrong venv shape and so
could not have caught this; rebuilt it from the real installer layout
(ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed
a_wheel_runtime_with_no_registry_record_is_still_found fails against the
old fallback before passing against the fix. (3) probe_comgr,
comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so
the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the
selection, and the multi-copy note could all break with the rest of the
suite still green. Split the ordering/selection logic into a pure
find_comgr_copies, the ldconfig -p line parsing into
parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings
(opt directory taken as a parameter rather than hardcoded, so a test can
point it at a temp folder instead of the real /opt on the machine running
the suite); each is unit-tested directly, and each new test was confirmed
to fail against a deliberately broken version of the function it covers
before being restored. (4) This message previously claimed a HIP copy
list and a single shared description of the loader path launched
processes use; neither matches this diff. Corrected above.

Review: a second round, one automated and one human, reaching the same
place by different routes.

The human reviewer confirmed, independently, that the site_packages:
None fallback added in the prior round could not actually fire:
ROCM_SDK_PROBE_SCRIPT has recorded site_packages unconditionally, outside
its try, since the probe's initial version, so every manifest that
parses at all carries Some there. The test built to cover it could not
have caught that either way, since it called the function directly with
None rather than through a real candidate. Rather than keep a reachable-
looking branch and a comment describing behaviour the code does not
have, the None arm is now a no-op (root's own directories, already
collected unconditionally above, are what a hand-constructed None caller
still gets) and the test is removed. The neighbouring root-format test
is kept and re-targeted: it already called the same function directly
with None, so it is what actually exercises that arm's contract, not the
removed test.

The automated reviewer caught a real regression the first round
introduced: three new unit tests feed a path into the `:`-split
LD_LIBRARY_PATH parser without a unix cfg gate, same as the pattern the
file already uses for that reason, and that broke the required
windows-build-and-test lane. Gated, along with the helper they share
(which would otherwise be unused, and clippy-denied, once its only
callers are gated).

It also found that the active-runtime-priority fix regresses
examine-finds-the-managed-runtimes-own-compilation-library on a healthy
host: once a managed runtime is active, its directory is found first and
labelled active-runtime, and the dedup this search has always done on
resolved path then drops the later managed-runtime hit for the identical
file -- the scenario's step asserted on the label that fix makes
unreachable. The step now accepts either label as proof the CLI's own
installed copy was found, and a new unit test
(an_active_runtime_directory_that_is_also_a_managed_root_keeps_one_label)
pins that an overlapping directory keeps exactly one label rather than
being reported twice; confirmed to fail when the active-runtime search is
moved after the others, then restored.

Review: a third round, three issues.

(1) comgr_matches_runtime was declared, defaulted, and listed in the
expected-keys test, but nothing in this diff computes it -- comparing
comgr's copies against the HIP runtime's is still deferred, same as the
commit message already said. A reader of the JSON contract alone had no
way to know the field was permanently unset rather than merely not yet
exercised by a test. Dropped from this commit entirely (the struct field,
its Default entry, its expected-keys assertion); `serde(default)` makes
re-adding it free once something actually computes it.

(2) comgr_selected named the first entry in comgr_paths as "would load"
regardless of which tier found it, so a machine carrying only a leftover
/opt/rocm-* install or an inactive managed runtime -- never a copy on
active-runtime, ld-library-path, or loader-cache -- was told that stale
copy would load, when the loader does not search /opt or a managed
runtime's directory on its own. find_comgr_copies now selects only from
those three loader-consulted tiers (LOADER_PATH_SOURCES), leaving
comgr_selected None and rewording the note when every hit is
rocm-install/managed-runtime-only; pinned by
a_search_dir_only_hit_is_not_reported_as_the_one_that_would_load. The e2e
step (assert_comgr_selection_is_stated) now asserts the same tier rule
against the real JSON: selected absent plus only non-loader-tier copies
is the pass case, not a shape to tolerate.

(3) comgr_search_dirs and managed_runtime_roots had no unit tests of
their own, and swapping the loader-cache and search-dir loops inside
find_comgr_copies left every then-existing test green -- nothing pinned
their relative order. Split comgr_search_dirs into a pure
comgr_search_dirs_in (rocm_path, the /opt directory, and the resolved
managed runtimes all taken as parameters) and added
comgr_search_dirs_labels_each_tier, covering all three source labels
against temp directories. Added loader_cache_outranks_search_dirs, built
so an ld-library-path hit cannot mask the order under test. Confirmed the
claim directly: swapped the two loops, reran the suite -- exactly
loader_cache_outranks_search_dirs failed, the other 65 tests (including
ld_library_path_outranks_loader_cache_and_search_dirs) stayed green --
then reverted the swap and reran clean before committing.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
jussielo-amd pushed a commit to jussielo-amd/rocm-cli that referenced this pull request Oct 6, 2026
… first (ROCm#384)

* feat(examine): report every code object manager library, not just the first

HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.

The search stopped at the first match, so the second copy could not be
seen at all. It now records every copy found, in loader search order.
Attributing each copy to the installation that owns it, and doing the
same search for the HIP runtime so the two can be compared, is deferred;
comgr_matches_runtime stays permanently unset until that lands.

The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why `managed_sdk_ld_library_path` (the
environment the CLI builds to run an SDK tool, such as
rocm_agent_enumerator, inside a managed runtime) walks that directory.
Searching the root alone therefore missed the copy this CLI installs
itself, which is the case the entry exists for.

That knowledge now lives in one place for this search:
collect_managed_runtime_library_paths is shared by the comgr search here
and by managed_sdk_ld_library_path, so that one layout is described once
there rather than twice. A third description, collect_runtime_environment_paths,
still builds the environment a served engine process gets; unifying all
three is follow-up work, not part of this search.

Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.

The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record, root's own `_rocm_sdk_*` name is read
to find its parent -- the site-packages directory its siblings share --
and a miss costs a directory that is simply not reported.

No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.

Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.

The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.

Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.

The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.

Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.

Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, ROCm#444), then examine-17 (EAI-8449,
ROCm#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.

Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.

Review: four more issues. (1) comgr_selected and the multi-copy note named
the ld.so.cache system copy as "would load" whenever a managed wheel
runtime was active but this process's own LD_LIBRARY_PATH was empty --
wrong, because rocm serve/rocm chat prepend the active runtime's own
library directories onto LD_LIBRARY_PATH before launching the engine, so
the wheel copy is what a served process actually loads. probe_comgr now
takes the active runtime's interpreter (the same FrameworkInterpreter
already threaded through probe_framework) and searches its library_paths
first, ahead of the ambient environment; pinned by
the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The
site_packages: None fallback in collect_managed_runtime_library_paths
guessed a venv layout (root/lib/<python>/site-packages) that the installer
never produces -- root is one of the runtime's own _rocm_sdk_* package
directories, not a venv root, so the guess always found nothing. It now
reads root.parent() instead, which is the site-packages directory the
guessed layout could never reach. The test fixture
(wheel_runtime_on_disk) was built from the same wrong venv shape and so
could not have caught this; rebuilt it from the real installer layout
(ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed
a_wheel_runtime_with_no_registry_record_is_still_found fails against the
old fallback before passing against the fix. (3) probe_comgr,
comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so
the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the
selection, and the multi-copy note could all break with the rest of the
suite still green. Split the ordering/selection logic into a pure
find_comgr_copies, the ldconfig -p line parsing into
parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings
(opt directory taken as a parameter rather than hardcoded, so a test can
point it at a temp folder instead of the real /opt on the machine running
the suite); each is unit-tested directly, and each new test was confirmed
to fail against a deliberately broken version of the function it covers
before being restored. (4) This message previously claimed a HIP copy
list and a single shared description of the loader path launched
processes use; neither matches this diff. Corrected above.

Review: a second round, one automated and one human, reaching the same
place by different routes.

The human reviewer confirmed, independently, that the site_packages:
None fallback added in the prior round could not actually fire:
ROCM_SDK_PROBE_SCRIPT has recorded site_packages unconditionally, outside
its try, since the probe's initial version, so every manifest that
parses at all carries Some there. The test built to cover it could not
have caught that either way, since it called the function directly with
None rather than through a real candidate. Rather than keep a reachable-
looking branch and a comment describing behaviour the code does not
have, the None arm is now a no-op (root's own directories, already
collected unconditionally above, are what a hand-constructed None caller
still gets) and the test is removed. The neighbouring root-format test
is kept and re-targeted: it already called the same function directly
with None, so it is what actually exercises that arm's contract, not the
removed test.

The automated reviewer caught a real regression the first round
introduced: three new unit tests feed a path into the `:`-split
LD_LIBRARY_PATH parser without a unix cfg gate, same as the pattern the
file already uses for that reason, and that broke the required
windows-build-and-test lane. Gated, along with the helper they share
(which would otherwise be unused, and clippy-denied, once its only
callers are gated).

It also found that the active-runtime-priority fix regresses
examine-finds-the-managed-runtimes-own-compilation-library on a healthy
host: once a managed runtime is active, its directory is found first and
labelled active-runtime, and the dedup this search has always done on
resolved path then drops the later managed-runtime hit for the identical
file -- the scenario's step asserted on the label that fix makes
unreachable. The step now accepts either label as proof the CLI's own
installed copy was found, and a new unit test
(an_active_runtime_directory_that_is_also_a_managed_root_keeps_one_label)
pins that an overlapping directory keeps exactly one label rather than
being reported twice; confirmed to fail when the active-runtime search is
moved after the others, then restored.

Review: a third round, three issues.

(1) comgr_matches_runtime was declared, defaulted, and listed in the
expected-keys test, but nothing in this diff computes it -- comparing
comgr's copies against the HIP runtime's is still deferred, same as the
commit message already said. A reader of the JSON contract alone had no
way to know the field was permanently unset rather than merely not yet
exercised by a test. Dropped from this commit entirely (the struct field,
its Default entry, its expected-keys assertion); `serde(default)` makes
re-adding it free once something actually computes it.

(2) comgr_selected named the first entry in comgr_paths as "would load"
regardless of which tier found it, so a machine carrying only a leftover
/opt/rocm-* install or an inactive managed runtime -- never a copy on
active-runtime, ld-library-path, or loader-cache -- was told that stale
copy would load, when the loader does not search /opt or a managed
runtime's directory on its own. find_comgr_copies now selects only from
those three loader-consulted tiers (LOADER_PATH_SOURCES), leaving
comgr_selected None and rewording the note when every hit is
rocm-install/managed-runtime-only; pinned by
a_search_dir_only_hit_is_not_reported_as_the_one_that_would_load. The e2e
step (assert_comgr_selection_is_stated) now asserts the same tier rule
against the real JSON: selected absent plus only non-loader-tier copies
is the pass case, not a shape to tolerate.

(3) comgr_search_dirs and managed_runtime_roots had no unit tests of
their own, and swapping the loader-cache and search-dir loops inside
find_comgr_copies left every then-existing test green -- nothing pinned
their relative order. Split comgr_search_dirs into a pure
comgr_search_dirs_in (rocm_path, the /opt directory, and the resolved
managed runtimes all taken as parameters) and added
comgr_search_dirs_labels_each_tier, covering all three source labels
against temp directories. Added loader_cache_outranks_search_dirs, built
so an ld-library-path hit cannot mask the order under test. Confirmed the
claim directly: swapped the two loops, reran the suite -- exactly
loader_cache_outranks_search_dirs failed, the other 65 tests (including
ld_library_path_outranks_loader_cache_and_search_dirs) stayed green --
then reverted the swap and reran clean before committing.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>

* feat(diagnose): report a code object manager that is not the runtime's own (ROCm#385)

Builds on the copy search: with every copy of libamd_comgr known, the
catalog can say when the one that loads belongs to a different
installation than the HIP runtime that loads, and device code
compilation therefore fails with an error naming neither.

The rule is symmetric, and that is what makes it safe. The design note
for this entry proposed a special case -- "when the active runtime is
the managed runtime, the matching copy is the wheel copy" -- without
which it "fires on every healthy CLI installation". That is a patch over
a rule stated asymmetrically. Asking one question of both libraries
instead, does the libamd_comgr that would load come from the same
installation as the libamdhip64 that would load, makes every case fall
out of the rule: a healthy managed install takes both from the managed
runtime, so nothing differs and nothing is reported. A control that has
to be written as an exception is a rule that has not been stated
correctly yet.

That turns on attributing a copy to its installation correctly. A
managed runtime spreads its libraries across separate _rocm_sdk_*
packages that sit beside each other under one site-packages, not nested
under the runtime's own root at all, so ownership is decided by looking a
found file's directory up against the same directories
install_library_dirs already recorded for each root -- never by walking
up from the file, which would call each package its own install and fire
on the most common install we ship. That is the property the mutation
check targets.

Two further guards on firing: an unattributed copy is a gap in what the
search knows rather than a finding about the machine, and the runtime
must actually ship a copy of its own, or there is nothing to point the
user at.

The entry is print-only and ranks neither remedy. Removing a stack and
reordering the search path can each break a working Python environment,
and which is right depends on which stack the user means to keep.

The finding says out loud that it describes the environment outside the
managed runtimes. `rocm serve` puts a managed runtime's libraries first
on purpose and gets a different, correct answer, and a report that did
not name the environment it examined would read as a claim about one it
never looked at.

The rule above is now stated exactly once, as `comgr_matches_runtime` in
examine.rs, and both this entry and `rocm examine --json`'s field of the
same name call it rather than each restating the conditions -- so the
two surfaces cannot disagree about one machine, and a regression test
pins that. Fix option (b)'s advice matched an earlier, asymmetric
statement of the rule; it now says to prefer the wheel's own copy of
both libraries, which is what the symmetric rule actually asks the user
to do. The machine-readable inspection now carries the same
`install_root` attribution for the HIP runtime side that it already
carried for the code object manager side, since the conflict question is
about both libraries and a reader could not previously check either the
attribution or the selection for one of them.

Known roots are deduplicated across the whole list, not only where
repeats happen to land next to each other. `dedup_by` drops consecutive
duplicates, and these roots arrive from three independent sources -- the
active install, a sorted /opt scan, and the managed runtimes -- so the
active install colliding with a sibling was caught or missed depending on
where the path sorted. A root recorded twice can make a machine holding
one stack read as a machine holding two, which is the false conflict this
entry exists not to report.

hip_paths and hip_selected are `serde(default)` for the same reason the
comgr fields are: an examination read back over the remote path may have
been produced by an older CLI that never wrote them.

Once a managed runtime is found, its root resolves to its
`_rocm_sdk_devel` package directory, and the search used to guess a venv
layout (`root/lib/<python>/site-packages`) to find its sibling packages;
that guess does not hold for a real install, so the search now reads
back the library directories the SDK probe already recorded
(`library_paths`) instead of re-deriving them. This generalises the
walker introduced in ROCm#384 (`find_library_copies`/`install_library_dirs`)
so it serves the HIP side too, which is why `managed_runtime_roots` now
returns `library_paths` plural rather than a single `site_packages`
guess.

The AppPaths::discover data-dir fix and the @requires-os:linux platform
tags this commit originally carried for examine-17/18 belong with the
scenario they correct, which ROCm#384 introduced -- moved there so the
parent doesn't depend on the child to pass its own CI, and this commit
now only rebases on top of that fix rather than restating it.

Sweeping the same gap once it was visible: examine-17 (the
host-independent code-object-manager scenario, renumbered by ROCm#384's
rebase) asserts `hip_paths`/`hip_selected` the same way it asserts
`comgr_paths`/`comgr_selected`, and the empty branch had the identical
problem ROCm#384's review flagged for the comgr side -- `hip_paths: []` /
`hip_selected: null` also hold by nothing more than `Examination`'s own
defaults, so the assertion could not tell "probed, found none" from
"never probed". `probe_comgr` now pushes a "no libamdhip64 found" note
the same way it already did for libamd_comgr, and the HIP assertion now
requires it in the empty branch. Verified by temporarily suppressing the
note and watching the scenario go red, then restoring it.

Review: the attribution half still did not survive a realistic managed
runtime. `known_install_roots` registers a managed runtime by
`candidate.root_path` alone -- `_rocm_sdk_devel` when the `devel` extra
is installed -- and `libamd_comgr`/`libamdhip64` live in the sibling
`_rocm_sdk_core` package, which sits *beside* `root_path`, not under it.
A string-prefix match against the bare root therefore attributed nothing
on exactly this shape: both libraries came back `install_root: ""`,
`comgr_matches_runtime` hit its empty-root guard and returned `None`
instead of `true`, and `check_18_comgr_conflict` could not fire for that
runtime at all. The two unit tests meant to catch this
(`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime`,
`a_healthy_managed_installation_raises_no_report`) built their fixtures
from a venv root (`/data/runtimes/therock/default`,
`.../lib/python3.12/site-packages/_rocm_sdk_core/...`) that production
never produces, so they passed regardless.

Fixed at the root: `install_library_dirs` already walks, for every known
root, exactly the directories that root's copies live in (that is the
discovery-side fix this commit already made). `find_library_copies` now
builds a directory -> owning-root map from that same walk and attributes
each found file by looking its directory up in it, rather than by
string-prefix match against the bare root. A sibling package's directory
is simply one more entry in the map, keyed to the runtime that recorded
it, so it attributes correctly without being nested under anything.
`owning_install_root` and `record_library_copy` take that map instead of
the bare root list; `the_nearest_enclosing_installation_claims_a_library`
is removed because the "longest prefix wins" ambiguity it guarded against
cannot arise under exact directory lookup. Both fixture tests above were
rebuilt from the real installer shape (sibling `_rocm_sdk_*` packages
under one `site-packages`, matching `apps/rocm/src/therock.rs`'s
`ROCM_SDK_PROBE_SCRIPT`), and
`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` was
confirmed to fail against the old string-prefix attribution before
passing against the fix.

`find_library_copies` also gained the `active_runtime_dirs` parameter
runtime's own library directories are now searched first, ahead of the
ambient `LD_LIBRARY_PATH`, for both comgr and HIP, since `rocm serve`
prepends them for both libraries equally. The loader cache and the
directory walk are now gathered once in `probe_comgr` and passed to both
the comgr and HIP searches, rather than each search reading
`LD_LIBRARY_PATH`, spawning `ldconfig -p`, and walking every managed
runtime's directories again on every invocation -- neither depends on
which library is being searched for, so the duplication cost an extra
subprocess and directory walk on every `rocm examine`/`rocm diagnose`
for no reason. `multi_copy_note` is split out as its own pure function
so the "N copies ... would load" note stays independently testable now
that `find_library_copies` only returns copies.

Two doc-comment splices, both from a missing blank `///` line between two
unrelated items: `check_18_comgr_conflict`'s doc paragraph had absorbed
check_17's above it, leaving check_17 undocumented; the same shape left
`known_install_roots` undocumented under a paragraph describing
`dedup_roots_keeping_first`. Both are split back onto the function they
actually describe.

`docs/wsl.md` said `fix-19-shm-too-small` was the only non-WSL entry that
applies on WSL; registering `check_18_comgr_conflict` for `["linux",
"wsl"]` made that false the moment this entry existed. Updated to name
both.

The "Neither option is recommended..." sentence was stated independently
in `fix.rs`'s static recipe and `diagnose.rs`'s dynamically-generated one,
in slightly different words -- already drifted, and only the `fix.rs`
copy was pinned by an e2e assertion, so the `diagnose.rs` copy could drift
further with nothing to catch it. Both now read
`fix::COMGR_CONFLICT_NEITHER_OPTION_RECOMMENDED`, so they cannot
disagree.

Not changed: `fix.rs`'s option (b) wording and `comgr_matches_runtime`'s
parity with `check_18_comgr_conflict` were flagged in an earlier review
round and were already correct by the time this round started --
verified against the current code rather than re-fixed.

Rebase onto ROCm#384 and a second symmetry gap: ROCm#384's own review fixed
`comgr_selected` picking `copies.first()` regardless of tier -- a
`rocm-install`/`managed-runtime`-only hit used to be reported as "the
one that would load" purely because it sorted first, when neither
source is one the loader consults on its own. Rebasing that fix under
this entry's HIP-runtime work surfaced a second instance of the same
asymmetry a human reviewer caught directly on this branch:
`probe_comgr`'s doc comment claimed comgr and HIP were searched and
reported identically, but only comgr ever got a `multi_copy_note` call;
the HIP branch only checked `hip.is_empty()` and pushed nothing for the
"found, but only in a directory the loader does not consult" case the
comgr side had just been fixed to report.

Fixed by stating the tier rule once and applying it to both prefixes
rather than fixing it twice: `select_loader_copy` replaces
`copies.first()` with a search restricted to `LOADER_PATH_SOURCES`
(`active-runtime`, `ld-library-path`, `loader-cache` -- the tiers the
loader itself consults), and `copies_note` replaces `multi_copy_note`
with the three-way note (no copies / more than one with a winner /
one-or-more with none selected) that a `rocm-install`-only hit now
needs. `probe_comgr` calls both exactly once for comgr and once for HIP,
so there is one call site per library rather than one function's logic
duplicated by hand into the other's branch, and the doc comment now says
what the code does instead of what it used to do before ROCm#384's fix
landed.

Pinned by `a_search_dir_only_hit_is_not_selected_for_either_prefix`,
which runs the same assertions for `COMGR_LIB_PREFIX` and
`HIP_RUNTIME_LIB_PREFIX` in one test rather than two, so the rule is
proven to apply to both rather than merely written to apply to both.
The existing `loader_cache_outranks_search_dirs` and
`the_active_runtimes_own_copy_outranks_the_ambient_environment`/
`ld_library_path_outranks_loader_cache_and_search_dirs` tests were
carried through the rebase onto `find_library_copies`/
`select_loader_copy` rather than dropped, since they pin mutation-tested
tier-ordering bugs ROCm#384 already fixed once.

`comgr_search_dirs_labels_each_tier`, which tested the now-deleted
`comgr_search_dirs_in`, is removed as redundant: `install_library_dirs`'s
own tier-labelling is already covered by
`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` and
`a_library_no_installation_claims_is_attributed_to_none`.

Verified: `cargo fmt --all -- --check`, `cargo clippy --workspace
--all-targets -- -D warnings`, `cargo clippy -p e2e-cucumber --test e2e
-- -D warnings`, and `cargo test --workspace --all-targets` all pass
(one `xtask::workflow_contract` test flaked under full-suite parallel
load on a timing assertion against a real `rocm-smi` subprocess call and
passed cleanly in isolation and on a full-suite rerun; unrelated to this
change). Cross-compiled the whole workspace, tests included, for
`x86_64-pc-windows-gnu` under `-D warnings` to confirm no code is left
unix-only-reachable and therefore dead on Windows. `cargo xtask e2e --
-n skill` passes all 6 scenarios, since this round also touches
`skills/rocm-doctor/reference.md`.

Second review round, two independent reviewers filing the same four findings
(verified rather than merely accepted):

fix-18's evidence line claimed to describe "the environment as it stands
outside the CLI's managed runtimes" unconditionally, even when a selection
actually came from the active-runtime tier -- a served child's search order,
not a plain shell's. That also means the `export LD_LIBRARY_PATH=...` remedy
can be inert for exactly that case, since `lemonade_process_environment_vars`
prepends the runtime's own directories ahead of whatever the remedy exports.
Rewording the evidence line is the easy half; the detection logic that
decides which sentence to print cannot move to plain-shell-only without
reintroducing the false positives/negatives `comgr_matches_runtime` and
`a_healthy_managed_installation_raises_no_report` were written to rule out,
since that guarantee depends on both selections already being
active-runtime-tier-aware. So `comgr_matches_runtime` is untouched;
`check_18_comgr_conflict` now checks whether either selected copy's source is
`active-runtime` and, when it is, swaps in an honest sentence plus a new
remedy note stating the export may not reach a served child and pointing at
reinstalling/repairing that runtime instead. Pinned by
`an_active_runtime_s_own_hip_without_its_own_comgr_notes_the_scope_limit`,
mutation-tested by forcing the branch condition false and watching the new
test fail on the stale sentence, then reverting.

`assert_hip_selection_is_stated`, the e2e HIP twin of the comgr selection
assertion, had only two branches, so "copies found, but none on a tier the
loader consults" -- `select_loader_copy` returning `None` on purpose, already
pinned for `libamdhip64` by
`a_search_dir_only_hit_is_not_selected_for_either_prefix` -- fell into the
`else` arm and asserted against a selected copy that does not exist. Added
the missing third branch, mirroring `assert_comgr_selection_is_stated`
exactly, including its `LOADER_PATH_SOURCES` check on the selected copy's
source, which the HIP step's final arm was also missing. Hoisted the shared
`LOADER_PATH_SOURCES` const the two steps now both use so they cannot drift
apart again.

`skills/rocm-doctor/reference.md`'s "Closed catalog (24 failure modes)"
heading and `SKILL.md`'s "the other 20 are print-only" line had both gone
stale against the table's 25 rows and 4 auto-applicable fixes
(`exactly_the_four_known_fixes_are_auto`); fixed to 25 and 21.
`rocm_doctor_skill.feature` deliberately leaves both counts unparsed against
the live CLI, and that scope decision stands -- the CLI is not wired to
report "how many rows exist" as a fact an agent reads. What it does not rule
out is checking the two numbers against each other, so
`tests/e2e-cucumber/tests/skill_reference.rs` (plain `cargo test`, no binary
needed) gained `free_standing_catalog_counts_match_the_table`, which parses
both numbers back out of the two docs and asserts them against the table's
own row and auto-fix counts. Mutation-tested by reverting each number one at
a time and confirming the new test goes red, then restoring it.

Open question from the first round, never answered: does
`install_library_dirs` calling `collect_sdk_library_paths` unconditionally
widen what a bare-metal `/opt/rocm` root probes in a way that matters?
Traced it rather than taking the earlier answer on faith: every directory
`install_library_dirs` adds for a root carries that root's own source
(`rocm-install` or `managed-runtime`, never a loader-consulted tier), and
`find_library_copies` only records a copy under that source when no
loader-consulted tier (`active-runtime`/`ld-library-path`/`loader-cache`)
already found the same path first -- so a wider probe can only add more
`rocm-install`-sourced evidence, never promote a path onto a tier
`select_loader_copy` would pick. The one place a wider probe does change an
answer is `comgr_matches_runtime`'s "does the runtime ship its own copy"
check, and there a new hit can only turn a false conflict report into `None`
(not enough evidence), which is strictly more correct, not a new risk. The
reviewer's reasoning holds; recorded here since the first round left it
unanswered.

Re-verified after this round's changes: `cargo fmt --all -- --check`, `cargo
clippy --workspace --all-targets -- -D warnings`, `cargo clippy -p
e2e-cucumber --test e2e -- -D warnings`, `cargo test --workspace
--all-targets` (251 passed, 0 failed across 33 binaries), and
`RUSTFLAGS="-D warnings" cargo check --workspace --all-targets --target
x86_64-pc-windows-gnu` all pass clean. `cargo xtask e2e -- -n skill` passes
all 6 scenarios again; `cargo xtask e2e -- -n examine-18` passes its 5 steps,
confirming the new HIP e2e branch does not break the existing scenario.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>

---------

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
sbmthakur added a commit to sbmthakur/rocm-cli that referenced this pull request Oct 12, 2026
The rocminfo walker kept a running agent name that every `Name:` line
overwrote, and a GPU agent lists its ISAs as further `Name:` lines
(`amdgcn-amd-amdhsa--gfx1100`, `...--gfx11-generic`) after `Device Type:`.
By the next `Agent` header the name no longer started with `gfx`, so every
GPU agent was dropped and rocminfo contributed nothing: no marketing name
for a card the PCI scan could not name, no `is_apu` from the target's
family. The per-GPU `gfx_target` that was empty when ROCm#393 was filed is now
filled by the kernel membership pass (ROCm#434); this fixes the parser so the
rest of rocminfo's reading reaches the report.

Take the first `Name:` after an `Agent` header as the agent's name and
ignore the rest, same for `Marketing Name:` and `BDFID:`. The loop is
lifted into `parse_rocminfo_gpu_agents`, and `apply_rocminfo_gpu_agents`
joins agents to entries by PCI address (the `BDFID` rendered in the
`dddd:bb:dd.f` form `pci_ids_match` already compares) instead of by
position, so rocminfo's and lspci's enumeration orders need not agree.
Only an agent with no usable `BDFID:` falls back to position, a fallback
never displaces an exact match, and an agent whose `BDFID:` matched nothing
becomes its own GPU rather than land on an unrelated card.

rocminfo never overwrites a `gfx_target` already present. The membership
pass stamps each entry with the kernel's native target by address before
rocminfo runs; rocminfo's reading is the same value normally and the
masqueraded one under `HSA_OVERRIDE_GFX_VERSION`, which the probe also
scrubs from rocminfo's environment so an override cannot be read back as
the hardware's gfx.

`is_apu` follows the target's family wherever the family table has a
verdict (`gfx_apu_family_verdict`: known APU, known discrete, or not
covered) and keeps the earlier name-based guess otherwise; the hipInfo path
on Windows uses the same rule. A Raphael iGPU (gfx1036, not in the table)
keeps lspci's APU verdict instead of flipping to discrete, and a GPU
appended with no entry to fill is left `None` when its family is unknown.

`rocm diagnose`'s two framework-arch checks read every AMD target. Now
that ROCm#434 fills the iGPU's target, an APU+dGPU desktop on upstream PyTorch
wheels is told to reinstall torch for a device fix-9 tells it to hide, and
the mirror check goes quiet. `serving_gfx_targets` judges the discrete
targets only when a host has both kinds of GPU, naming the integrated
target it left out, and falls back to every target when no discrete one is
known. Hosts with one kind are judged as before.

No new scenario: examine-17 already asserts every GPU's PCI address and
gfx target against the kernel topology, and the diagnose change only
diverges on an iGPU+dGPU host whose wheel misses the iGPU, which no CI lane
provides; that shape is covered by unit tests.

Fixes ROCm#393

Signed-off-by: Shubham Thakur <sbmthakur@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants