Repository navigation
feat(diagnose): report a code object manager that is not the runtime's own - #385
Conversation
7d0af16 to
9452104
Compare
87cbcac to
a1968d9
Compare
9452104 to
4e5fe59
Compare
a1968d9 to
22e9b4a
Compare
4e5fe59 to
6f617c8
Compare
22e9b4a to
5c39eee
Compare
6f617c8 to
09476f8
Compare
5c39eee to
f1a1b9b
Compare
jussielo-amd
left a comment
There was a problem hiding this comment.
This is a solid feature (surfacing comgr/HIP runtime mismatches), but I found a few issues that affect correctness of the diagnosis output and the fix advice text, plus a couple of secondary concerns. Left inline comments on the ones anchored to this diff; one test-coverage gap below since it points at a file this PR doesn't touch.
Untested field, tests/e2e-cucumber/tests/e2e/examine_steps.rs (not modified by this PR): the e2e assertion for comgr_paths entries still only requires ["path", "real_path", "version", "source"] and wasn't extended to require the new install_root field this PR adds to the wire format, and no e2e step checks hip_paths/hip_selected either. Right now only in-process unit tests exercise install_root — the actual CLI --json output for the field this design turns on is unverified end-to-end.
Requesting changes mainly for the fix-18 advice/data mismatch (correctness bug in user-facing guidance) and the comgr_matches_runtime vs diagnose disagreement — both are logic errors, not style nits.
| "# Then pick ONE of the following. They are alternatives, not steps.", | ||
| "# (a) Keep the system installation: remove or uninstall the wheel that", | ||
| "# supplies the second copy.", | ||
| "# (b) Keep the wheel: order the search path so the runtime's own copy", |
There was a problem hiding this comment.
Option (b) here says "Keep the wheel: order the search path so the runtime's own copy is found first", followed by an LD_LIBRARY_PATH command — but per this PR's own conflict definition (diagnose.rs's matching), "the runtime's own copy" is the copy belonging to the currently-active HIP runtime, which in the canonical wheel/system conflict is the system install, not the wheel. A user who wants to keep the wheel and follows this advice literally will export LD_LIBRARY_PATH to the system comgr directory instead — the opposite of the stated intent — and there's no obvious symptom afterward besides now picking up the system's libraries. Worth double-checking which path this recipe actually resolves to for option (b) and fixing the wording/command so it points at the wheel's own comgr, not the runtime's.
| // `None` rather than `false` when either side is missing: "they disagree" and | ||
| // "there was nothing to compare" are different answers, and a caller that | ||
| // cannot tell them apart reports a conflict on a machine with no ROCm at all. | ||
| e.comgr_matches_runtime = match (comgr.first(), hip.first()) { |
There was a problem hiding this comment.
The documented --json field comgr_matches_runtime is computed with a weaker rule (comgr.install_root != hip.install_root, both non-empty) than check_18_comgr_conflict, which additionally requires the active runtime to ship a matching comgr copy of its own before firing. When the runtime ships no comgr copy of its own — the exact case this PR's own test a_runtime_with_no_copy_of_its_own_is_not_a_conflict covers — rocm examine --json will report comgr_matches_runtime: false while rocm diagnose reports nothing at all. Anyone consuming this field programmatically (or a user comparing the two commands) sees a contradiction. Suggest computing this field with the same logic check_18_comgr_conflict uses.
| let mut copies: Vec<LibraryCopy> = Vec::new(); | ||
| let mut seen: std::collections::BTreeSet<String> = std::collections::BTreeSet::new(); | ||
|
|
||
| // Order is the whole point: this is the order the loader consults, so the |
There was a problem hiding this comment.
find_library_copies recomputes install_library_dirs(roots) and re-invokes paths_in_loader_cache (which spawns ldconfig -p) once per prefix, so both now run twice per rocm diagnose/rocm examine call — once for comgr, once for hip — even though neither depends on which prefix is being searched. That's an extra ldconfig -p subprocess plus a second full walk of every managed runtime's lib/lib64 → pythonX/site-packages → _rocm_sdk_* directories on every invocation of a command that's typically run repeatedly while debugging. Might be worth hoisting the roots/loader-cache computation out so it's shared between the two calls.
| .into_iter() | ||
| .map(|root| (root, "managed-runtime")), | ||
| ); | ||
| roots.dedup_by(|a, b| a.0 == b.0); |
There was a problem hiding this comment.
known_install_roots dedups with Vec::dedup_by, which only removes adjacent duplicates. e.rocm_path is pushed first and the sorted /opt/* siblings are appended after, so if e.rocm_path equals one of those sorted entries but isn't adjacent to it in the final vector (e.g. e.rocm_path is /opt/rocm-6.4 and /opt also contains rocm and rocm-6.4, sorting to [rocm, rocm-6.4]), the roots vector ends up [rocm-6.4, rocm, rocm-6.4, ...] and the two rocm-6.4 entries survive the dedup. install_library_dirs/find_library_copies then scan that install's directories twice per probe — masked from the final output only because record_library_copy separately dedups by resolved real path downstream. Sorting the full vector (or using a HashSet) before dedup would close this.
f1a1b9b to
c591f8a
Compare
d9b0a75 to
f2bd0c9
Compare
c591f8a to
2a77f96
Compare
6ea3a21 to
ee48870
Compare
2a77f96 to
c6cfd42
Compare
ee48870 to
a8749a3
Compare
991f5d8 to
166d9ed
Compare
r0x0r
left a comment
There was a problem hiding this comment.
What the change does. Teaches examine/diagnose to notice that the libamd_comgr copy the loader would pick belongs to a different installation than the libamdhip64 runtime, reports the HIP copies alongside the comgr ones, and adds fix-18-comgr-conflict as an advisory-only entry that states both remedies without ranking them.
Coverage. Full read of crates/rocm-core/src/{examine,diagnose,fix,lib}.rs and the four e2e-cucumber files at head, reviewed against the PR's own base (feat/report-comgr-copies), so #384's changes are excluded. Checked against AGENTS.md §3. Both findings from the earlier, now-dismissed review were re-tested specifically. No tests run.
Assessment: looks sound — no blocking findings survived verification.
The earlier review's two findings are addressed
-
fix-18 advice/data mismatch — fixed. The machine-specific
Fixindiagnose.rsderives itsLD_LIBRARY_PATHdirectory frommatching.real_path, wherematchingis the comgr copy whoseinstall_rootequals the HIP runtime's, so "order the search path so the runtime's own copy is found first" now describes what the command actually does. The<directory of the wheel's own copy>placeholder that remains infix.rsis the generic catalog recipe, which has no live paths to fill in — correct for its role. -
comgr_matches_runtimevsdiagnosedisagreeing — fixed.probe_comgrandcheck_18_comgr_conflictnow call the sameexamine::comgr_matches_runtime, andcomgr_matches_runtime_never_disagrees_with_the_diagnosispins that invariant across the catalog cases, including the one that used to diverge (a runtime shipping no comgr copy of its own). Making the disagreement structurally impossible, rather than fixing the two sites to agree, is the right shape of fix. -
The e2e gap is closed too.
assert_comgr_copies_reportednow requiresinstall_root:for field in ["path", "real_path", "version", "source", "install_root"] {
and two new steps assert
hip_pathsandhip_selected, wired into examine-16.
AGENTS.md §3 is satisfied: diagnose-21 (@id:diagnose-fix-comgr-conflict-is-advisory-only) covers the user-observable half, and its comment explains honestly why the conflict itself cannot be provoked in the suite — the detection rule is left to unit tests that build machine state directly. That is the explanation §3 asks for rather than a silent omission.
Non-blocking
managed_runtime_roots()runs twice perfind_library_copies.known_install_roots(examine.rs:2170) calls it to buildroots, theninstall_library_dirs(:2200) calls it again to buildmanaged_library_paths, andprobe_comgrruns the pair twice (comgr, then HIP) — four registry walks per examination. It is a deterministic filesystem read, so this is cost rather than correctness, but threading the first result through would remove both the cost and any question about the two views agreeing.comgr_version_from_file_nameis now called for HIP paths too (record_library_copy). The soname parsing is correct for both; only the name still says comgr.assert_hip_selection_is_statedpasses trivially on the mock lane — withhip_paths == []it only assertship_selectedis null, so the list/selection agreement is never exercised off a GPU lane. It inherits this from the comgr sibling rather than introducing it.assert_fix_is_advisoryusescontains("print-only") || contains("will NOT run it"), so dropping either string still passes.assert_both_options_unrankedcovers the dangerous case, so this is slack rather than a hole.- Untested: the ordering contract in
find_library_copies(LD_LIBRARY_PATH beats loader cache beats install dirs) is the whole point of the function and has no unit test; neither does the partial-version evidence branch indiagnose.rswhere exactly one side has a version string.
Tradeoffs
comgr_matches_runtimereturnsNoneboth for "no known installation claims this copy" and for "the runtime ships no comgr of its own". Serialized, a consumer cannot tell those apart from "not computed". The comments document the distinction; the wire format does not carry it.- The symmetric rule ("do both come from the same installation?") stays quiet on a machine holding only the mismatched comgr, because there would be no alternative to point at. Deliberate and documented, but it does mean a genuinely broken single-stack install reports nothing.
Positive signals
- Driving the JSON field and the diagnostic from one function, then testing that they agree, removes a whole class of future drift.
- Longest-prefix attribution in
owning_install_roothandles managed runtimes spread across_rocm_sdk_*subdirectories without special cases, anddedup_roots_keeping_firsthas a test for the non-adjacent duplicate that a naive consecutive dedup would miss. auto_applicable: false, with both remedies stated and neither recommended, is the right posture for a fix where either choice can break a working Python environment.#[serde(default)]on every new field keeps older consumers reading the new output.
One thing outside the diff
This PR's base is #384, which currently has changes requested and 6 failing checks, so #385 cannot land ahead of it regardless of its own state. Its own checks are green at this head apart from one still running.
The merge decision is yours; this review is posted as a comment and files no approval or change request.
🤖 by agent-hub on AMD AgentHub
166d9ed to
1837ea8
Compare
a8749a3 to
51a20ff
Compare
51a20ff to
c5c15f7
Compare
4236280 to
19e0fda
Compare
c5c15f7 to
a6d7af6
Compare
juhovainio
left a comment
There was a problem hiding this comment.
I reviewed this PR and found one real gap plus a handful of smaller things, left inline below.
The gap: on a managed runtime installed with the devel extra, the new check can never fire, and comgr_matches_runtime comes back null rather than true on a healthy install. This PR fixes the discovery half of that problem, install_library_dirs now prefers the probe's recorded library_paths over re-deriving the layout, but the attribution half still compares those paths against a root that is the _rocm_sdk_devel directory, which they are not under. The unit tests do not catch it because their fixtures are built from a venv root that production never supplies.
The rest are documentation and description accuracy: two doc comments have drifted onto the wrong functions, docs/wsl.md now contradicts the registration table, and the description's CI section describes work that moved to #384.
One concern I chased and dropped: the claim that the venv-layout fallback had disappeared from examine's search. It has not, library_paths has been in the probe script since the initial import.
a6d7af6 to
e33f8ee
Compare
19e0fda to
ec761a5
Compare
ba1d7fb to
53b13d2
Compare
ec761a5 to
3dfb896
Compare
All five items addressed: attribution fixed from prefix-match to an exact directory-owner map (dir_owners), with the regression fixture rebuilt from the production sibling-directory layout and confirmed by mutation testing (reverted to the old prefix match, watched the test go red, reverted back, confirmed green and no residual diff); both doc-comment splices fixed to sit on their own functions; docs/wsl.md updated so it no longer contradicts the fix-18 registration; PR description corrected for examine-19 naming and CI attribution to #384; and the fix.rs/diagnose.rs remedy text now shares one constant, structurally removing the drift the e2e pin could miss. Full verification bar is clean: cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings, cargo clippy -p e2e-cucumber --test e2e -- -D warnings, cargo test --workspace --all-targets (45 binaries, 0 failures), and cargo xtask e2e -- -n skill (6 scenarios, 22 steps, 0 failures).
3dfb896 to
da83dcf
Compare
juhovainio
left a comment
There was a problem hiding this comment.
Re-reviewed after the fixes for the last round — all 5 previously-flagged issues are genuinely fixed, including the owning_install_root attribution bug (this now does exact-directory attribution via dir_owners, not just discovery).
One new issue in the comgr/HIP generalization: probe_comgr documents the comgr and HIP searches as symmetric ("searched the same way, and for the same reason"), but the "N copies found; X would load" note is only emitted for comgr. The HIP branch only checks hip.is_empty() and never calls multi_copy_note for it, so a machine with multiple libamdhip64 copies gets no note, even though the equivalent comgr case does. Left as an inline comment below.
53b13d2 to
13ed5d4
Compare
da83dcf to
955608f
Compare
r0x0r
left a comment
There was a problem hiding this comment.
I took another look at this at 955608fd. Last time I looked (at an older commit) I said nothing blocking survived; this round turned up four things worth fixing before merge, left inline below.
Two are catalog bookkeeping that nothing in CI will catch, and they're the cheapest to fix: reference.md's heading still says 24 failure modes when the table now has 25 rows, and SKILL.md's "the other 20 are print-only" is now 21. You did correctly bump fix.rs's assert_eq!(count, 25) and update the OS recap line, so these are just the two surfaces with no assertion behind them — rocm_doctor_skill.feature says outright that both are unparsed and can go stale.
The other two are substantive:
- The new HIP e2e step panics on a state this PR deliberately introduces.
assert_hip_selection_is_statedhas only two branches, so "copies found, but none on a tier the loader consults" hits theelsearm and panics — even thoughselect_loader_copyreturnsNonefor exactly that case and your own unit test pins it forlibamdhip64specifically. The comgr twin right below it already has the third branch; the HIP copy just didn't get it. - fix-18's evidence line tells the user the finding describes their plain shell, but the selection it rests on is computed with the active managed runtime's directories searched first.
find_library_copies' own doc comment says that order "is not this process's search order; it is a served child's." That matters because the remedy you hand the user is a shellexport, which can't change an outcome measured in the served-child order.
A few things I chased and dropped, for what it's worth: owning_install_root attributes from the canonicalised real_path, so a /opt/rocm symlink resolves both libraries to the same owner; check_18_comgr_conflict and comgr_matches_runtime really do share one predicate now, so they can't contradict each other; and the generated export LD_LIBRARY_PATH=... in diagnose.rs does point at the runtime's own copy, matching the prose above it.
Separately — the five comments from the earlier, now-dismissed review never got replies, so I checked them against this head. Four are genuinely fixed (option (b) wording, the comgr_matches_runtime rule, the doubled ldconfig -p, and the non-adjacent root dedup, which now has a test using the exact ordering from the comment). The fifth is still open: install_library_dirs calls collect_sdk_library_paths unconditionally, so bare-metal /opt/rocm roots now also get bin and lib/rocm_sysdeps/lib probed where the old comgr_search_dirs only added lib/lib64. I don't think it's a defect — those hits land on the rocm-install tier, which select_loader_copy now excludes from "would load" — but it was asked as a question and still deserves an answer in the PR text.
13ed5d4 to
e7d40a3
Compare
juhovainio
left a comment
There was a problem hiding this comment.
I independently verified r0x0r's four open findings against the current diff — all four are real and still unaddressed, so I'm requesting changes to confirm them rather than leave them resting only on r0x0r's earlier review.
Separately: the earlier multi_copy_note/HIP-branch asymmetry I flagged is fixed, and fixed the way Eugene described — probe_comgr's HIP arm now calls the same note/selection path as the comgr arm, and the incidental .first() tier-selection bug is gone from production code too. No new issues beyond what's below.
The one I'd treat as the real blocker is the diagnose.rs:1498 one: the printed evidence line claims to describe "the environment as it stands outside the CLI's managed runtimes," but the conflict this check fires on can happen with an active managed runtime (the runtime ships its own HIP but not its own comgr). In that case the evidence may have come from the active-runtime tier, not the plain-shell tiers the sentence claims — and the fix's export LD_LIBRARY_PATH=... advice doesn't touch what rocm serve actually loads, so it could be inert for exactly the case it's describing.
…s own Builds on the copy search: with every copy of libamd_comgr known, the catalog can say when the one that loads belongs to a different installation than the HIP runtime that loads, and device code compilation therefore fails with an error naming neither. The rule is symmetric, and that is what makes it safe. The design note for this entry proposed a special case -- "when the active runtime is the managed runtime, the matching copy is the wheel copy" -- without which it "fires on every healthy CLI installation". That is a patch over a rule stated asymmetrically. Asking one question of both libraries instead, does the libamd_comgr that would load come from the same installation as the libamdhip64 that would load, makes every case fall out of the rule: a healthy managed install takes both from the managed runtime, so nothing differs and nothing is reported. A control that has to be written as an exception is a rule that has not been stated correctly yet. That turns on attributing a copy to its installation correctly. A managed runtime spreads its libraries across separate _rocm_sdk_* packages that sit beside each other under one site-packages, not nested under the runtime's own root at all, so ownership is decided by looking a found file's directory up against the same directories install_library_dirs already recorded for each root -- never by walking up from the file, which would call each package its own install and fire on the most common install we ship. That is the property the mutation check targets. Two further guards on firing: an unattributed copy is a gap in what the search knows rather than a finding about the machine, and the runtime must actually ship a copy of its own, or there is nothing to point the user at. The entry is print-only and ranks neither remedy. Removing a stack and reordering the search path can each break a working Python environment, and which is right depends on which stack the user means to keep. The finding says out loud that it describes the environment outside the managed runtimes. `rocm serve` puts a managed runtime's libraries first on purpose and gets a different, correct answer, and a report that did not name the environment it examined would read as a claim about one it never looked at. The rule above is now stated exactly once, as `comgr_matches_runtime` in examine.rs, and both this entry and `rocm examine --json`'s field of the same name call it rather than each restating the conditions -- so the two surfaces cannot disagree about one machine, and a regression test pins that. Fix option (b)'s advice matched an earlier, asymmetric statement of the rule; it now says to prefer the wheel's own copy of both libraries, which is what the symmetric rule actually asks the user to do. The machine-readable inspection now carries the same `install_root` attribution for the HIP runtime side that it already carried for the code object manager side, since the conflict question is about both libraries and a reader could not previously check either the attribution or the selection for one of them. Known roots are deduplicated across the whole list, not only where repeats happen to land next to each other. `dedup_by` drops consecutive duplicates, and these roots arrive from three independent sources -- the active install, a sorted /opt scan, and the managed runtimes -- so the active install colliding with a sibling was caught or missed depending on where the path sorted. A root recorded twice can make a machine holding one stack read as a machine holding two, which is the false conflict this entry exists not to report. hip_paths and hip_selected are `serde(default)` for the same reason the comgr fields are: an examination read back over the remote path may have been produced by an older CLI that never wrote them. Once a managed runtime is found, its root resolves to its `_rocm_sdk_devel` package directory, and the search used to guess a venv layout (`root/lib/<python>/site-packages`) to find its sibling packages; that guess does not hold for a real install, so the search now reads back the library directories the SDK probe already recorded (`library_paths`) instead of re-deriving them. This generalises the walker introduced in #384 (`find_library_copies`/`install_library_dirs`) so it serves the HIP side too, which is why `managed_runtime_roots` now returns `library_paths` plural rather than a single `site_packages` guess. The AppPaths::discover data-dir fix and the @requires-os:linux platform tags this commit originally carried for examine-17/18 belong with the scenario they correct, which #384 introduced -- moved there so the parent doesn't depend on the child to pass its own CI, and this commit now only rebases on top of that fix rather than restating it. Sweeping the same gap once it was visible: examine-17 (the host-independent code-object-manager scenario, renumbered by #384's rebase) asserts `hip_paths`/`hip_selected` the same way it asserts `comgr_paths`/`comgr_selected`, and the empty branch had the identical problem #384's review flagged for the comgr side -- `hip_paths: []` / `hip_selected: null` also hold by nothing more than `Examination`'s own defaults, so the assertion could not tell "probed, found none" from "never probed". `probe_comgr` now pushes a "no libamdhip64 found" note the same way it already did for libamd_comgr, and the HIP assertion now requires it in the empty branch. Verified by temporarily suppressing the note and watching the scenario go red, then restoring it. Review: the attribution half still did not survive a realistic managed runtime. `known_install_roots` registers a managed runtime by `candidate.root_path` alone -- `_rocm_sdk_devel` when the `devel` extra is installed -- and `libamd_comgr`/`libamdhip64` live in the sibling `_rocm_sdk_core` package, which sits *beside* `root_path`, not under it. A string-prefix match against the bare root therefore attributed nothing on exactly this shape: both libraries came back `install_root: ""`, `comgr_matches_runtime` hit its empty-root guard and returned `None` instead of `true`, and `check_18_comgr_conflict` could not fire for that runtime at all. The two unit tests meant to catch this (`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime`, `a_healthy_managed_installation_raises_no_report`) built their fixtures from a venv root (`/data/runtimes/therock/default`, `.../lib/python3.12/site-packages/_rocm_sdk_core/...`) that production never produces, so they passed regardless. Fixed at the root: `install_library_dirs` already walks, for every known root, exactly the directories that root's copies live in (that is the discovery-side fix this commit already made). `find_library_copies` now builds a directory -> owning-root map from that same walk and attributes each found file by looking its directory up in it, rather than by string-prefix match against the bare root. A sibling package's directory is simply one more entry in the map, keyed to the runtime that recorded it, so it attributes correctly without being nested under anything. `owning_install_root` and `record_library_copy` take that map instead of the bare root list; `the_nearest_enclosing_installation_claims_a_library` is removed because the "longest prefix wins" ambiguity it guarded against cannot arise under exact directory lookup. Both fixture tests above were rebuilt from the real installer shape (sibling `_rocm_sdk_*` packages under one `site-packages`, matching `apps/rocm/src/therock.rs`'s `ROCM_SDK_PROBE_SCRIPT`), and `a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` was confirmed to fail against the old string-prefix attribution before passing against the fix. `find_library_copies` also gained the `active_runtime_dirs` parameter runtime's own library directories are now searched first, ahead of the ambient `LD_LIBRARY_PATH`, for both comgr and HIP, since `rocm serve` prepends them for both libraries equally. The loader cache and the directory walk are now gathered once in `probe_comgr` and passed to both the comgr and HIP searches, rather than each search reading `LD_LIBRARY_PATH`, spawning `ldconfig -p`, and walking every managed runtime's directories again on every invocation -- neither depends on which library is being searched for, so the duplication cost an extra subprocess and directory walk on every `rocm examine`/`rocm diagnose` for no reason. `multi_copy_note` is split out as its own pure function so the "N copies ... would load" note stays independently testable now that `find_library_copies` only returns copies. Two doc-comment splices, both from a missing blank `///` line between two unrelated items: `check_18_comgr_conflict`'s doc paragraph had absorbed check_17's above it, leaving check_17 undocumented; the same shape left `known_install_roots` undocumented under a paragraph describing `dedup_roots_keeping_first`. Both are split back onto the function they actually describe. `docs/wsl.md` said `fix-19-shm-too-small` was the only non-WSL entry that applies on WSL; registering `check_18_comgr_conflict` for `["linux", "wsl"]` made that false the moment this entry existed. Updated to name both. The "Neither option is recommended..." sentence was stated independently in `fix.rs`'s static recipe and `diagnose.rs`'s dynamically-generated one, in slightly different words -- already drifted, and only the `fix.rs` copy was pinned by an e2e assertion, so the `diagnose.rs` copy could drift further with nothing to catch it. Both now read `fix::COMGR_CONFLICT_NEITHER_OPTION_RECOMMENDED`, so they cannot disagree. Not changed: `fix.rs`'s option (b) wording and `comgr_matches_runtime`'s parity with `check_18_comgr_conflict` were flagged in an earlier review round and were already correct by the time this round started -- verified against the current code rather than re-fixed. Rebase onto #384 and a second symmetry gap: #384's own review fixed `comgr_selected` picking `copies.first()` regardless of tier -- a `rocm-install`/`managed-runtime`-only hit used to be reported as "the one that would load" purely because it sorted first, when neither source is one the loader consults on its own. Rebasing that fix under this entry's HIP-runtime work surfaced a second instance of the same asymmetry a human reviewer caught directly on this branch: `probe_comgr`'s doc comment claimed comgr and HIP were searched and reported identically, but only comgr ever got a `multi_copy_note` call; the HIP branch only checked `hip.is_empty()` and pushed nothing for the "found, but only in a directory the loader does not consult" case the comgr side had just been fixed to report. Fixed by stating the tier rule once and applying it to both prefixes rather than fixing it twice: `select_loader_copy` replaces `copies.first()` with a search restricted to `LOADER_PATH_SOURCES` (`active-runtime`, `ld-library-path`, `loader-cache` -- the tiers the loader itself consults), and `copies_note` replaces `multi_copy_note` with the three-way note (no copies / more than one with a winner / one-or-more with none selected) that a `rocm-install`-only hit now needs. `probe_comgr` calls both exactly once for comgr and once for HIP, so there is one call site per library rather than one function's logic duplicated by hand into the other's branch, and the doc comment now says what the code does instead of what it used to do before #384's fix landed. Pinned by `a_search_dir_only_hit_is_not_selected_for_either_prefix`, which runs the same assertions for `COMGR_LIB_PREFIX` and `HIP_RUNTIME_LIB_PREFIX` in one test rather than two, so the rule is proven to apply to both rather than merely written to apply to both. The existing `loader_cache_outranks_search_dirs` and `the_active_runtimes_own_copy_outranks_the_ambient_environment`/ `ld_library_path_outranks_loader_cache_and_search_dirs` tests were carried through the rebase onto `find_library_copies`/ `select_loader_copy` rather than dropped, since they pin mutation-tested tier-ordering bugs #384 already fixed once. `comgr_search_dirs_labels_each_tier`, which tested the now-deleted `comgr_search_dirs_in`, is removed as redundant: `install_library_dirs`'s own tier-labelling is already covered by `a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` and `a_library_no_installation_claims_is_attributed_to_none`. Verified: `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo clippy -p e2e-cucumber --test e2e -- -D warnings`, and `cargo test --workspace --all-targets` all pass (one `xtask::workflow_contract` test flaked under full-suite parallel load on a timing assertion against a real `rocm-smi` subprocess call and passed cleanly in isolation and on a full-suite rerun; unrelated to this change). Cross-compiled the whole workspace, tests included, for `x86_64-pc-windows-gnu` under `-D warnings` to confirm no code is left unix-only-reachable and therefore dead on Windows. `cargo xtask e2e -- -n skill` passes all 6 scenarios, since this round also touches `skills/rocm-doctor/reference.md`. Second review round, two independent reviewers filing the same four findings (verified rather than merely accepted): fix-18's evidence line claimed to describe "the environment as it stands outside the CLI's managed runtimes" unconditionally, even when a selection actually came from the active-runtime tier -- a served child's search order, not a plain shell's. That also means the `export LD_LIBRARY_PATH=...` remedy can be inert for exactly that case, since `lemonade_process_environment_vars` prepends the runtime's own directories ahead of whatever the remedy exports. Rewording the evidence line is the easy half; the detection logic that decides which sentence to print cannot move to plain-shell-only without reintroducing the false positives/negatives `comgr_matches_runtime` and `a_healthy_managed_installation_raises_no_report` were written to rule out, since that guarantee depends on both selections already being active-runtime-tier-aware. So `comgr_matches_runtime` is untouched; `check_18_comgr_conflict` now checks whether either selected copy's source is `active-runtime` and, when it is, swaps in an honest sentence plus a new remedy note stating the export may not reach a served child and pointing at reinstalling/repairing that runtime instead. Pinned by `an_active_runtime_s_own_hip_without_its_own_comgr_notes_the_scope_limit`, mutation-tested by forcing the branch condition false and watching the new test fail on the stale sentence, then reverting. `assert_hip_selection_is_stated`, the e2e HIP twin of the comgr selection assertion, had only two branches, so "copies found, but none on a tier the loader consults" -- `select_loader_copy` returning `None` on purpose, already pinned for `libamdhip64` by `a_search_dir_only_hit_is_not_selected_for_either_prefix` -- fell into the `else` arm and asserted against a selected copy that does not exist. Added the missing third branch, mirroring `assert_comgr_selection_is_stated` exactly, including its `LOADER_PATH_SOURCES` check on the selected copy's source, which the HIP step's final arm was also missing. Hoisted the shared `LOADER_PATH_SOURCES` const the two steps now both use so they cannot drift apart again. `skills/rocm-doctor/reference.md`'s "Closed catalog (24 failure modes)" heading and `SKILL.md`'s "the other 20 are print-only" line had both gone stale against the table's 25 rows and 4 auto-applicable fixes (`exactly_the_four_known_fixes_are_auto`); fixed to 25 and 21. `rocm_doctor_skill.feature` deliberately leaves both counts unparsed against the live CLI, and that scope decision stands -- the CLI is not wired to report "how many rows exist" as a fact an agent reads. What it does not rule out is checking the two numbers against each other, so `tests/e2e-cucumber/tests/skill_reference.rs` (plain `cargo test`, no binary needed) gained `free_standing_catalog_counts_match_the_table`, which parses both numbers back out of the two docs and asserts them against the table's own row and auto-fix counts. Mutation-tested by reverting each number one at a time and confirming the new test goes red, then restoring it. Open question from the first round, never answered: does `install_library_dirs` calling `collect_sdk_library_paths` unconditionally widen what a bare-metal `/opt/rocm` root probes in a way that matters? Traced it rather than taking the earlier answer on faith: every directory `install_library_dirs` adds for a root carries that root's own source (`rocm-install` or `managed-runtime`, never a loader-consulted tier), and `find_library_copies` only records a copy under that source when no loader-consulted tier (`active-runtime`/`ld-library-path`/`loader-cache`) already found the same path first -- so a wider probe can only add more `rocm-install`-sourced evidence, never promote a path onto a tier `select_loader_copy` would pick. The one place a wider probe does change an answer is `comgr_matches_runtime`'s "does the runtime ship its own copy" check, and there a new hit can only turn a false conflict report into `None` (not enough evidence), which is strictly more correct, not a new risk. The reviewer's reasoning holds; recorded here since the first round left it unanswered. Re-verified after this round's changes: `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo clippy -p e2e-cucumber --test e2e -- -D warnings`, `cargo test --workspace --all-targets` (251 passed, 0 failed across 33 binaries), and `RUSTFLAGS="-D warnings" cargo check --workspace --all-targets --target x86_64-pc-windows-gnu` all pass clean. `cargo xtask e2e -- -n skill` passes all 6 scenarios again; `cargo xtask e2e -- -n examine-18` passes its 5 steps, confirming the new HIP e2e branch does not break the existing scenario. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
955608f to
0ed84ba
Compare
All four findings addressed in the amended commit: fix-18's evidence/remedy made conditional on active-runtime involvement (not just reworded), the HIP e2e step's missing third branch added (unified with the comgr step via select_loader_copy/copies_note), reference.md/SKILL.md counts fixed to 25/21 with a new drift-guard test, and the install_library_dirs scan-widening question traced and confirmed safe. Replied inline to each; see the PR description's "Review remediation, round two" section.
|
Addressed the second review round (r0x0r + juhovainio, same four findings) in fix-18 evidence/remedy (#4184271400, #4185096435) — made the evidence line conditional on whether either selected copy came from the HIP e2e missing third branch (#4184271391, #4185096422) — unified the comgr/HIP selection+note logic ( Stale catalog counts (#4184271375, #4184271383, #4185096413) — fixed to 25/21, and added Open question on Full detail is in the commit message's "Review remediation, round two" section and the PR description. Re-ran the full verification bar plus |
juhovainio
left a comment
There was a problem hiding this comment.
Checked all three fixes against the actual current code, not just the replies:
diagnose.rs's comgr-conflict evidence line is now conditional on whether either selected copy came from the active-runtime tier, with the honest sentence either way plus an added fix note that theexportmay not reach a served child. Pinned by a new unit test.examine_steps.rs's HIP-selection assertion now has the missing third branch for "copies found, none selected" (select_loader_copyreturningNoneon purpose), matching the comgr sibling.- The catalog heading and
SKILL.md's print-only count both now match the live table (25 rows, 4 auto-applicable = 21), backed by a newcargo test-level drift guard so this can't go stale again silently.
Looks good to merge.
…s own (#385) Builds on the copy search: with every copy of libamd_comgr known, the catalog can say when the one that loads belongs to a different installation than the HIP runtime that loads, and device code compilation therefore fails with an error naming neither. The rule is symmetric, and that is what makes it safe. The design note for this entry proposed a special case -- "when the active runtime is the managed runtime, the matching copy is the wheel copy" -- without which it "fires on every healthy CLI installation". That is a patch over a rule stated asymmetrically. Asking one question of both libraries instead, does the libamd_comgr that would load come from the same installation as the libamdhip64 that would load, makes every case fall out of the rule: a healthy managed install takes both from the managed runtime, so nothing differs and nothing is reported. A control that has to be written as an exception is a rule that has not been stated correctly yet. That turns on attributing a copy to its installation correctly. A managed runtime spreads its libraries across separate _rocm_sdk_* packages that sit beside each other under one site-packages, not nested under the runtime's own root at all, so ownership is decided by looking a found file's directory up against the same directories install_library_dirs already recorded for each root -- never by walking up from the file, which would call each package its own install and fire on the most common install we ship. That is the property the mutation check targets. Two further guards on firing: an unattributed copy is a gap in what the search knows rather than a finding about the machine, and the runtime must actually ship a copy of its own, or there is nothing to point the user at. The entry is print-only and ranks neither remedy. Removing a stack and reordering the search path can each break a working Python environment, and which is right depends on which stack the user means to keep. The finding says out loud that it describes the environment outside the managed runtimes. `rocm serve` puts a managed runtime's libraries first on purpose and gets a different, correct answer, and a report that did not name the environment it examined would read as a claim about one it never looked at. The rule above is now stated exactly once, as `comgr_matches_runtime` in examine.rs, and both this entry and `rocm examine --json`'s field of the same name call it rather than each restating the conditions -- so the two surfaces cannot disagree about one machine, and a regression test pins that. Fix option (b)'s advice matched an earlier, asymmetric statement of the rule; it now says to prefer the wheel's own copy of both libraries, which is what the symmetric rule actually asks the user to do. The machine-readable inspection now carries the same `install_root` attribution for the HIP runtime side that it already carried for the code object manager side, since the conflict question is about both libraries and a reader could not previously check either the attribution or the selection for one of them. Known roots are deduplicated across the whole list, not only where repeats happen to land next to each other. `dedup_by` drops consecutive duplicates, and these roots arrive from three independent sources -- the active install, a sorted /opt scan, and the managed runtimes -- so the active install colliding with a sibling was caught or missed depending on where the path sorted. A root recorded twice can make a machine holding one stack read as a machine holding two, which is the false conflict this entry exists not to report. hip_paths and hip_selected are `serde(default)` for the same reason the comgr fields are: an examination read back over the remote path may have been produced by an older CLI that never wrote them. Once a managed runtime is found, its root resolves to its `_rocm_sdk_devel` package directory, and the search used to guess a venv layout (`root/lib/<python>/site-packages`) to find its sibling packages; that guess does not hold for a real install, so the search now reads back the library directories the SDK probe already recorded (`library_paths`) instead of re-deriving them. This generalises the walker introduced in #384 (`find_library_copies`/`install_library_dirs`) so it serves the HIP side too, which is why `managed_runtime_roots` now returns `library_paths` plural rather than a single `site_packages` guess. The AppPaths::discover data-dir fix and the @requires-os:linux platform tags this commit originally carried for examine-17/18 belong with the scenario they correct, which #384 introduced -- moved there so the parent doesn't depend on the child to pass its own CI, and this commit now only rebases on top of that fix rather than restating it. Sweeping the same gap once it was visible: examine-17 (the host-independent code-object-manager scenario, renumbered by #384's rebase) asserts `hip_paths`/`hip_selected` the same way it asserts `comgr_paths`/`comgr_selected`, and the empty branch had the identical problem #384's review flagged for the comgr side -- `hip_paths: []` / `hip_selected: null` also hold by nothing more than `Examination`'s own defaults, so the assertion could not tell "probed, found none" from "never probed". `probe_comgr` now pushes a "no libamdhip64 found" note the same way it already did for libamd_comgr, and the HIP assertion now requires it in the empty branch. Verified by temporarily suppressing the note and watching the scenario go red, then restoring it. Review: the attribution half still did not survive a realistic managed runtime. `known_install_roots` registers a managed runtime by `candidate.root_path` alone -- `_rocm_sdk_devel` when the `devel` extra is installed -- and `libamd_comgr`/`libamdhip64` live in the sibling `_rocm_sdk_core` package, which sits *beside* `root_path`, not under it. A string-prefix match against the bare root therefore attributed nothing on exactly this shape: both libraries came back `install_root: ""`, `comgr_matches_runtime` hit its empty-root guard and returned `None` instead of `true`, and `check_18_comgr_conflict` could not fire for that runtime at all. The two unit tests meant to catch this (`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime`, `a_healthy_managed_installation_raises_no_report`) built their fixtures from a venv root (`/data/runtimes/therock/default`, `.../lib/python3.12/site-packages/_rocm_sdk_core/...`) that production never produces, so they passed regardless. Fixed at the root: `install_library_dirs` already walks, for every known root, exactly the directories that root's copies live in (that is the discovery-side fix this commit already made). `find_library_copies` now builds a directory -> owning-root map from that same walk and attributes each found file by looking its directory up in it, rather than by string-prefix match against the bare root. A sibling package's directory is simply one more entry in the map, keyed to the runtime that recorded it, so it attributes correctly without being nested under anything. `owning_install_root` and `record_library_copy` take that map instead of the bare root list; `the_nearest_enclosing_installation_claims_a_library` is removed because the "longest prefix wins" ambiguity it guarded against cannot arise under exact directory lookup. Both fixture tests above were rebuilt from the real installer shape (sibling `_rocm_sdk_*` packages under one `site-packages`, matching `apps/rocm/src/therock.rs`'s `ROCM_SDK_PROBE_SCRIPT`), and `a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` was confirmed to fail against the old string-prefix attribution before passing against the fix. `find_library_copies` also gained the `active_runtime_dirs` parameter runtime's own library directories are now searched first, ahead of the ambient `LD_LIBRARY_PATH`, for both comgr and HIP, since `rocm serve` prepends them for both libraries equally. The loader cache and the directory walk are now gathered once in `probe_comgr` and passed to both the comgr and HIP searches, rather than each search reading `LD_LIBRARY_PATH`, spawning `ldconfig -p`, and walking every managed runtime's directories again on every invocation -- neither depends on which library is being searched for, so the duplication cost an extra subprocess and directory walk on every `rocm examine`/`rocm diagnose` for no reason. `multi_copy_note` is split out as its own pure function so the "N copies ... would load" note stays independently testable now that `find_library_copies` only returns copies. Two doc-comment splices, both from a missing blank `///` line between two unrelated items: `check_18_comgr_conflict`'s doc paragraph had absorbed check_17's above it, leaving check_17 undocumented; the same shape left `known_install_roots` undocumented under a paragraph describing `dedup_roots_keeping_first`. Both are split back onto the function they actually describe. `docs/wsl.md` said `fix-19-shm-too-small` was the only non-WSL entry that applies on WSL; registering `check_18_comgr_conflict` for `["linux", "wsl"]` made that false the moment this entry existed. Updated to name both. The "Neither option is recommended..." sentence was stated independently in `fix.rs`'s static recipe and `diagnose.rs`'s dynamically-generated one, in slightly different words -- already drifted, and only the `fix.rs` copy was pinned by an e2e assertion, so the `diagnose.rs` copy could drift further with nothing to catch it. Both now read `fix::COMGR_CONFLICT_NEITHER_OPTION_RECOMMENDED`, so they cannot disagree. Not changed: `fix.rs`'s option (b) wording and `comgr_matches_runtime`'s parity with `check_18_comgr_conflict` were flagged in an earlier review round and were already correct by the time this round started -- verified against the current code rather than re-fixed. Rebase onto #384 and a second symmetry gap: #384's own review fixed `comgr_selected` picking `copies.first()` regardless of tier -- a `rocm-install`/`managed-runtime`-only hit used to be reported as "the one that would load" purely because it sorted first, when neither source is one the loader consults on its own. Rebasing that fix under this entry's HIP-runtime work surfaced a second instance of the same asymmetry a human reviewer caught directly on this branch: `probe_comgr`'s doc comment claimed comgr and HIP were searched and reported identically, but only comgr ever got a `multi_copy_note` call; the HIP branch only checked `hip.is_empty()` and pushed nothing for the "found, but only in a directory the loader does not consult" case the comgr side had just been fixed to report. Fixed by stating the tier rule once and applying it to both prefixes rather than fixing it twice: `select_loader_copy` replaces `copies.first()` with a search restricted to `LOADER_PATH_SOURCES` (`active-runtime`, `ld-library-path`, `loader-cache` -- the tiers the loader itself consults), and `copies_note` replaces `multi_copy_note` with the three-way note (no copies / more than one with a winner / one-or-more with none selected) that a `rocm-install`-only hit now needs. `probe_comgr` calls both exactly once for comgr and once for HIP, so there is one call site per library rather than one function's logic duplicated by hand into the other's branch, and the doc comment now says what the code does instead of what it used to do before #384's fix landed. Pinned by `a_search_dir_only_hit_is_not_selected_for_either_prefix`, which runs the same assertions for `COMGR_LIB_PREFIX` and `HIP_RUNTIME_LIB_PREFIX` in one test rather than two, so the rule is proven to apply to both rather than merely written to apply to both. The existing `loader_cache_outranks_search_dirs` and `the_active_runtimes_own_copy_outranks_the_ambient_environment`/ `ld_library_path_outranks_loader_cache_and_search_dirs` tests were carried through the rebase onto `find_library_copies`/ `select_loader_copy` rather than dropped, since they pin mutation-tested tier-ordering bugs #384 already fixed once. `comgr_search_dirs_labels_each_tier`, which tested the now-deleted `comgr_search_dirs_in`, is removed as redundant: `install_library_dirs`'s own tier-labelling is already covered by `a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` and `a_library_no_installation_claims_is_attributed_to_none`. Verified: `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo clippy -p e2e-cucumber --test e2e -- -D warnings`, and `cargo test --workspace --all-targets` all pass (one `xtask::workflow_contract` test flaked under full-suite parallel load on a timing assertion against a real `rocm-smi` subprocess call and passed cleanly in isolation and on a full-suite rerun; unrelated to this change). Cross-compiled the whole workspace, tests included, for `x86_64-pc-windows-gnu` under `-D warnings` to confirm no code is left unix-only-reachable and therefore dead on Windows. `cargo xtask e2e -- -n skill` passes all 6 scenarios, since this round also touches `skills/rocm-doctor/reference.md`. Second review round, two independent reviewers filing the same four findings (verified rather than merely accepted): fix-18's evidence line claimed to describe "the environment as it stands outside the CLI's managed runtimes" unconditionally, even when a selection actually came from the active-runtime tier -- a served child's search order, not a plain shell's. That also means the `export LD_LIBRARY_PATH=...` remedy can be inert for exactly that case, since `lemonade_process_environment_vars` prepends the runtime's own directories ahead of whatever the remedy exports. Rewording the evidence line is the easy half; the detection logic that decides which sentence to print cannot move to plain-shell-only without reintroducing the false positives/negatives `comgr_matches_runtime` and `a_healthy_managed_installation_raises_no_report` were written to rule out, since that guarantee depends on both selections already being active-runtime-tier-aware. So `comgr_matches_runtime` is untouched; `check_18_comgr_conflict` now checks whether either selected copy's source is `active-runtime` and, when it is, swaps in an honest sentence plus a new remedy note stating the export may not reach a served child and pointing at reinstalling/repairing that runtime instead. Pinned by `an_active_runtime_s_own_hip_without_its_own_comgr_notes_the_scope_limit`, mutation-tested by forcing the branch condition false and watching the new test fail on the stale sentence, then reverting. `assert_hip_selection_is_stated`, the e2e HIP twin of the comgr selection assertion, had only two branches, so "copies found, but none on a tier the loader consults" -- `select_loader_copy` returning `None` on purpose, already pinned for `libamdhip64` by `a_search_dir_only_hit_is_not_selected_for_either_prefix` -- fell into the `else` arm and asserted against a selected copy that does not exist. Added the missing third branch, mirroring `assert_comgr_selection_is_stated` exactly, including its `LOADER_PATH_SOURCES` check on the selected copy's source, which the HIP step's final arm was also missing. Hoisted the shared `LOADER_PATH_SOURCES` const the two steps now both use so they cannot drift apart again. `skills/rocm-doctor/reference.md`'s "Closed catalog (24 failure modes)" heading and `SKILL.md`'s "the other 20 are print-only" line had both gone stale against the table's 25 rows and 4 auto-applicable fixes (`exactly_the_four_known_fixes_are_auto`); fixed to 25 and 21. `rocm_doctor_skill.feature` deliberately leaves both counts unparsed against the live CLI, and that scope decision stands -- the CLI is not wired to report "how many rows exist" as a fact an agent reads. What it does not rule out is checking the two numbers against each other, so `tests/e2e-cucumber/tests/skill_reference.rs` (plain `cargo test`, no binary needed) gained `free_standing_catalog_counts_match_the_table`, which parses both numbers back out of the two docs and asserts them against the table's own row and auto-fix counts. Mutation-tested by reverting each number one at a time and confirming the new test goes red, then restoring it. Open question from the first round, never answered: does `install_library_dirs` calling `collect_sdk_library_paths` unconditionally widen what a bare-metal `/opt/rocm` root probes in a way that matters? Traced it rather than taking the earlier answer on faith: every directory `install_library_dirs` adds for a root carries that root's own source (`rocm-install` or `managed-runtime`, never a loader-consulted tier), and `find_library_copies` only records a copy under that source when no loader-consulted tier (`active-runtime`/`ld-library-path`/`loader-cache`) already found the same path first -- so a wider probe can only add more `rocm-install`-sourced evidence, never promote a path onto a tier `select_loader_copy` would pick. The one place a wider probe does change an answer is `comgr_matches_runtime`'s "does the runtime ship its own copy" check, and there a new hit can only turn a false conflict report into `None` (not enough evidence), which is strictly more correct, not a new risk. The reviewer's reasoning holds; recorded here since the first round left it unanswered. Re-verified after this round's changes: `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo clippy -p e2e-cucumber --test e2e -- -D warnings`, `cargo test --workspace --all-targets` (251 passed, 0 failed across 33 binaries), and `RUSTFLAGS="-D warnings" cargo check --workspace --all-targets --target x86_64-pc-windows-gnu` all pass clean. `cargo xtask e2e -- -n skill` passes all 6 scenarios again; `cargo xtask e2e -- -n examine-18` passes its 5 steps, confirming the new HIP e2e branch does not break the existing scenario. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
…s own (#385) Builds on the copy search: with every copy of libamd_comgr known, the catalog can say when the one that loads belongs to a different installation than the HIP runtime that loads, and device code compilation therefore fails with an error naming neither. The rule is symmetric, and that is what makes it safe. The design note for this entry proposed a special case -- "when the active runtime is the managed runtime, the matching copy is the wheel copy" -- without which it "fires on every healthy CLI installation". That is a patch over a rule stated asymmetrically. Asking one question of both libraries instead, does the libamd_comgr that would load come from the same installation as the libamdhip64 that would load, makes every case fall out of the rule: a healthy managed install takes both from the managed runtime, so nothing differs and nothing is reported. A control that has to be written as an exception is a rule that has not been stated correctly yet. That turns on attributing a copy to its installation correctly. A managed runtime spreads its libraries across separate _rocm_sdk_* packages that sit beside each other under one site-packages, not nested under the runtime's own root at all, so ownership is decided by looking a found file's directory up against the same directories install_library_dirs already recorded for each root -- never by walking up from the file, which would call each package its own install and fire on the most common install we ship. That is the property the mutation check targets. Two further guards on firing: an unattributed copy is a gap in what the search knows rather than a finding about the machine, and the runtime must actually ship a copy of its own, or there is nothing to point the user at. The entry is print-only and ranks neither remedy. Removing a stack and reordering the search path can each break a working Python environment, and which is right depends on which stack the user means to keep. The finding says out loud that it describes the environment outside the managed runtimes. `rocm serve` puts a managed runtime's libraries first on purpose and gets a different, correct answer, and a report that did not name the environment it examined would read as a claim about one it never looked at. The rule above is now stated exactly once, as `comgr_matches_runtime` in examine.rs, and both this entry and `rocm examine --json`'s field of the same name call it rather than each restating the conditions -- so the two surfaces cannot disagree about one machine, and a regression test pins that. Fix option (b)'s advice matched an earlier, asymmetric statement of the rule; it now says to prefer the wheel's own copy of both libraries, which is what the symmetric rule actually asks the user to do. The machine-readable inspection now carries the same `install_root` attribution for the HIP runtime side that it already carried for the code object manager side, since the conflict question is about both libraries and a reader could not previously check either the attribution or the selection for one of them. Known roots are deduplicated across the whole list, not only where repeats happen to land next to each other. `dedup_by` drops consecutive duplicates, and these roots arrive from three independent sources -- the active install, a sorted /opt scan, and the managed runtimes -- so the active install colliding with a sibling was caught or missed depending on where the path sorted. A root recorded twice can make a machine holding one stack read as a machine holding two, which is the false conflict this entry exists not to report. hip_paths and hip_selected are `serde(default)` for the same reason the comgr fields are: an examination read back over the remote path may have been produced by an older CLI that never wrote them. Once a managed runtime is found, its root resolves to its `_rocm_sdk_devel` package directory, and the search used to guess a venv layout (`root/lib/<python>/site-packages`) to find its sibling packages; that guess does not hold for a real install, so the search now reads back the library directories the SDK probe already recorded (`library_paths`) instead of re-deriving them. This generalises the walker introduced in #384 (`find_library_copies`/`install_library_dirs`) so it serves the HIP side too, which is why `managed_runtime_roots` now returns `library_paths` plural rather than a single `site_packages` guess. The AppPaths::discover data-dir fix and the @requires-os:linux platform tags this commit originally carried for examine-17/18 belong with the scenario they correct, which #384 introduced -- moved there so the parent doesn't depend on the child to pass its own CI, and this commit now only rebases on top of that fix rather than restating it. Sweeping the same gap once it was visible: examine-17 (the host-independent code-object-manager scenario, renumbered by #384's rebase) asserts `hip_paths`/`hip_selected` the same way it asserts `comgr_paths`/`comgr_selected`, and the empty branch had the identical problem #384's review flagged for the comgr side -- `hip_paths: []` / `hip_selected: null` also hold by nothing more than `Examination`'s own defaults, so the assertion could not tell "probed, found none" from "never probed". `probe_comgr` now pushes a "no libamdhip64 found" note the same way it already did for libamd_comgr, and the HIP assertion now requires it in the empty branch. Verified by temporarily suppressing the note and watching the scenario go red, then restoring it. Review: the attribution half still did not survive a realistic managed runtime. `known_install_roots` registers a managed runtime by `candidate.root_path` alone -- `_rocm_sdk_devel` when the `devel` extra is installed -- and `libamd_comgr`/`libamdhip64` live in the sibling `_rocm_sdk_core` package, which sits *beside* `root_path`, not under it. A string-prefix match against the bare root therefore attributed nothing on exactly this shape: both libraries came back `install_root: ""`, `comgr_matches_runtime` hit its empty-root guard and returned `None` instead of `true`, and `check_18_comgr_conflict` could not fire for that runtime at all. The two unit tests meant to catch this (`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime`, `a_healthy_managed_installation_raises_no_report`) built their fixtures from a venv root (`/data/runtimes/therock/default`, `.../lib/python3.12/site-packages/_rocm_sdk_core/...`) that production never produces, so they passed regardless. Fixed at the root: `install_library_dirs` already walks, for every known root, exactly the directories that root's copies live in (that is the discovery-side fix this commit already made). `find_library_copies` now builds a directory -> owning-root map from that same walk and attributes each found file by looking its directory up in it, rather than by string-prefix match against the bare root. A sibling package's directory is simply one more entry in the map, keyed to the runtime that recorded it, so it attributes correctly without being nested under anything. `owning_install_root` and `record_library_copy` take that map instead of the bare root list; `the_nearest_enclosing_installation_claims_a_library` is removed because the "longest prefix wins" ambiguity it guarded against cannot arise under exact directory lookup. Both fixture tests above were rebuilt from the real installer shape (sibling `_rocm_sdk_*` packages under one `site-packages`, matching `apps/rocm/src/therock.rs`'s `ROCM_SDK_PROBE_SCRIPT`), and `a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` was confirmed to fail against the old string-prefix attribution before passing against the fix. `find_library_copies` also gained the `active_runtime_dirs` parameter runtime's own library directories are now searched first, ahead of the ambient `LD_LIBRARY_PATH`, for both comgr and HIP, since `rocm serve` prepends them for both libraries equally. The loader cache and the directory walk are now gathered once in `probe_comgr` and passed to both the comgr and HIP searches, rather than each search reading `LD_LIBRARY_PATH`, spawning `ldconfig -p`, and walking every managed runtime's directories again on every invocation -- neither depends on which library is being searched for, so the duplication cost an extra subprocess and directory walk on every `rocm examine`/`rocm diagnose` for no reason. `multi_copy_note` is split out as its own pure function so the "N copies ... would load" note stays independently testable now that `find_library_copies` only returns copies. Two doc-comment splices, both from a missing blank `///` line between two unrelated items: `check_18_comgr_conflict`'s doc paragraph had absorbed check_17's above it, leaving check_17 undocumented; the same shape left `known_install_roots` undocumented under a paragraph describing `dedup_roots_keeping_first`. Both are split back onto the function they actually describe. `docs/wsl.md` said `fix-19-shm-too-small` was the only non-WSL entry that applies on WSL; registering `check_18_comgr_conflict` for `["linux", "wsl"]` made that false the moment this entry existed. Updated to name both. The "Neither option is recommended..." sentence was stated independently in `fix.rs`'s static recipe and `diagnose.rs`'s dynamically-generated one, in slightly different words -- already drifted, and only the `fix.rs` copy was pinned by an e2e assertion, so the `diagnose.rs` copy could drift further with nothing to catch it. Both now read `fix::COMGR_CONFLICT_NEITHER_OPTION_RECOMMENDED`, so they cannot disagree. Not changed: `fix.rs`'s option (b) wording and `comgr_matches_runtime`'s parity with `check_18_comgr_conflict` were flagged in an earlier review round and were already correct by the time this round started -- verified against the current code rather than re-fixed. Rebase onto #384 and a second symmetry gap: #384's own review fixed `comgr_selected` picking `copies.first()` regardless of tier -- a `rocm-install`/`managed-runtime`-only hit used to be reported as "the one that would load" purely because it sorted first, when neither source is one the loader consults on its own. Rebasing that fix under this entry's HIP-runtime work surfaced a second instance of the same asymmetry a human reviewer caught directly on this branch: `probe_comgr`'s doc comment claimed comgr and HIP were searched and reported identically, but only comgr ever got a `multi_copy_note` call; the HIP branch only checked `hip.is_empty()` and pushed nothing for the "found, but only in a directory the loader does not consult" case the comgr side had just been fixed to report. Fixed by stating the tier rule once and applying it to both prefixes rather than fixing it twice: `select_loader_copy` replaces `copies.first()` with a search restricted to `LOADER_PATH_SOURCES` (`active-runtime`, `ld-library-path`, `loader-cache` -- the tiers the loader itself consults), and `copies_note` replaces `multi_copy_note` with the three-way note (no copies / more than one with a winner / one-or-more with none selected) that a `rocm-install`-only hit now needs. `probe_comgr` calls both exactly once for comgr and once for HIP, so there is one call site per library rather than one function's logic duplicated by hand into the other's branch, and the doc comment now says what the code does instead of what it used to do before #384's fix landed. Pinned by `a_search_dir_only_hit_is_not_selected_for_either_prefix`, which runs the same assertions for `COMGR_LIB_PREFIX` and `HIP_RUNTIME_LIB_PREFIX` in one test rather than two, so the rule is proven to apply to both rather than merely written to apply to both. The existing `loader_cache_outranks_search_dirs` and `the_active_runtimes_own_copy_outranks_the_ambient_environment`/ `ld_library_path_outranks_loader_cache_and_search_dirs` tests were carried through the rebase onto `find_library_copies`/ `select_loader_copy` rather than dropped, since they pin mutation-tested tier-ordering bugs #384 already fixed once. `comgr_search_dirs_labels_each_tier`, which tested the now-deleted `comgr_search_dirs_in`, is removed as redundant: `install_library_dirs`'s own tier-labelling is already covered by `a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` and `a_library_no_installation_claims_is_attributed_to_none`. Verified: `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo clippy -p e2e-cucumber --test e2e -- -D warnings`, and `cargo test --workspace --all-targets` all pass (one `xtask::workflow_contract` test flaked under full-suite parallel load on a timing assertion against a real `rocm-smi` subprocess call and passed cleanly in isolation and on a full-suite rerun; unrelated to this change). Cross-compiled the whole workspace, tests included, for `x86_64-pc-windows-gnu` under `-D warnings` to confirm no code is left unix-only-reachable and therefore dead on Windows. `cargo xtask e2e -- -n skill` passes all 6 scenarios, since this round also touches `skills/rocm-doctor/reference.md`. Second review round, two independent reviewers filing the same four findings (verified rather than merely accepted): fix-18's evidence line claimed to describe "the environment as it stands outside the CLI's managed runtimes" unconditionally, even when a selection actually came from the active-runtime tier -- a served child's search order, not a plain shell's. That also means the `export LD_LIBRARY_PATH=...` remedy can be inert for exactly that case, since `lemonade_process_environment_vars` prepends the runtime's own directories ahead of whatever the remedy exports. Rewording the evidence line is the easy half; the detection logic that decides which sentence to print cannot move to plain-shell-only without reintroducing the false positives/negatives `comgr_matches_runtime` and `a_healthy_managed_installation_raises_no_report` were written to rule out, since that guarantee depends on both selections already being active-runtime-tier-aware. So `comgr_matches_runtime` is untouched; `check_18_comgr_conflict` now checks whether either selected copy's source is `active-runtime` and, when it is, swaps in an honest sentence plus a new remedy note stating the export may not reach a served child and pointing at reinstalling/repairing that runtime instead. Pinned by `an_active_runtime_s_own_hip_without_its_own_comgr_notes_the_scope_limit`, mutation-tested by forcing the branch condition false and watching the new test fail on the stale sentence, then reverting. `assert_hip_selection_is_stated`, the e2e HIP twin of the comgr selection assertion, had only two branches, so "copies found, but none on a tier the loader consults" -- `select_loader_copy` returning `None` on purpose, already pinned for `libamdhip64` by `a_search_dir_only_hit_is_not_selected_for_either_prefix` -- fell into the `else` arm and asserted against a selected copy that does not exist. Added the missing third branch, mirroring `assert_comgr_selection_is_stated` exactly, including its `LOADER_PATH_SOURCES` check on the selected copy's source, which the HIP step's final arm was also missing. Hoisted the shared `LOADER_PATH_SOURCES` const the two steps now both use so they cannot drift apart again. `skills/rocm-doctor/reference.md`'s "Closed catalog (24 failure modes)" heading and `SKILL.md`'s "the other 20 are print-only" line had both gone stale against the table's 25 rows and 4 auto-applicable fixes (`exactly_the_four_known_fixes_are_auto`); fixed to 25 and 21. `rocm_doctor_skill.feature` deliberately leaves both counts unparsed against the live CLI, and that scope decision stands -- the CLI is not wired to report "how many rows exist" as a fact an agent reads. What it does not rule out is checking the two numbers against each other, so `tests/e2e-cucumber/tests/skill_reference.rs` (plain `cargo test`, no binary needed) gained `free_standing_catalog_counts_match_the_table`, which parses both numbers back out of the two docs and asserts them against the table's own row and auto-fix counts. Mutation-tested by reverting each number one at a time and confirming the new test goes red, then restoring it. Open question from the first round, never answered: does `install_library_dirs` calling `collect_sdk_library_paths` unconditionally widen what a bare-metal `/opt/rocm` root probes in a way that matters? Traced it rather than taking the earlier answer on faith: every directory `install_library_dirs` adds for a root carries that root's own source (`rocm-install` or `managed-runtime`, never a loader-consulted tier), and `find_library_copies` only records a copy under that source when no loader-consulted tier (`active-runtime`/`ld-library-path`/`loader-cache`) already found the same path first -- so a wider probe can only add more `rocm-install`-sourced evidence, never promote a path onto a tier `select_loader_copy` would pick. The one place a wider probe does change an answer is `comgr_matches_runtime`'s "does the runtime ship its own copy" check, and there a new hit can only turn a false conflict report into `None` (not enough evidence), which is strictly more correct, not a new risk. The reviewer's reasoning holds; recorded here since the first round left it unanswered. Re-verified after this round's changes: `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo clippy -p e2e-cucumber --test e2e -- -D warnings`, `cargo test --workspace --all-targets` (251 passed, 0 failed across 33 binaries), and `RUSTFLAGS="-D warnings" cargo check --workspace --all-targets --target x86_64-pc-windows-gnu` all pass clean. `cargo xtask e2e -- -n skill` passes all 6 scenarios again; `cargo xtask e2e -- -n examine-18` passes its 5 steps, confirming the new HIP e2e branch does not break the existing scenario. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
… first (ROCm#384) * feat(examine): report every code object manager library, not just the first HIP compiles device code at run time through libamd_comgr, and a machine can hold more than one copy. The loader picks one. When the copy it picks does not belong to the active HIP runtime, compilation fails with an error naming neither the library nor the second copy. The search stopped at the first match, so the second copy could not be seen at all. It now records every copy found, in loader search order. Attributing each copy to the installation that owns it, and doing the same search for the HIP runtime so the two can be compared, is deferred; comgr_matches_runtime stays permanently unset until that lands. The managed search now reads the runtime registry rather than listing <data>/runtimes on disk. A directory listing yields a root and nothing else, and the root alone does not locate a wheel runtime's libraries -- those live in a sibling _rocm_sdk_* package inside the interpreter's site-packages, which is why `managed_sdk_ld_library_path` (the environment the CLI builds to run an SDK tool, such as rocm_agent_enumerator, inside a managed runtime) walks that directory. Searching the root alone therefore missed the copy this CLI installs itself, which is the case the entry exists for. That knowledge now lives in one place for this search: collect_managed_runtime_library_paths is shared by the comgr search here and by managed_sdk_ld_library_path, so that one layout is described once there rather than twice. A third description, collect_runtime_environment_paths, still builds the environment a served engine process gets; unifying all three is follow-up work, not part of this search. Listing the directory was also a second answer to "which runtimes exist", where the registry is the first one, and only the registry records which site-packages an interpreter uses. The record is preferred but not required. A read-only probe cannot depend on one being present and current, and a host whose registry is missing or stale is exactly the kind this command is called on: reporting no copies there would read as a machine with nothing wrong rather than one we failed to inspect. With no record, root's own `_rocm_sdk_*` name is read to find its parent -- the site-packages directory its siblings share -- and a miss costs a directory that is simply not reported. No dlopen. Reading the version through amd_comgr_get_version would run an unknown library's initialisers on a machine called on precisely because something is already wrong, and load a possibly conflicting HIP stack permanently into the process. The version is read from the versioned soname instead, and a name carrying none yields an empty version rather than a confident wrong one. Directory matches are now sorted, which makes the HIP probe's tie-break deterministic where one directory holds several matching files. It used to take whatever read_dir yielded first. Better, but a change, not a no-op. The loader-cache tier used to gate on which("ldconfig"), which only walks $PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian and derivatives. That silently dropped the one tier that finds a copy registered only in ld.so.cache; it now reuses the fallback search ldconfig_cache() already does for the same reason. Covered at two levels, because neither is sufficient alone. Unit tests build a wheel-format layout directly, which proves the search understands a layout we described; only a real managed runtime proves it matches the one the installer produces, so examine-19 asserts that on a GPU lane. The four new fields are `serde(default)`. This structure is read back from another machine: `rocm remote doctor` deserializes an examination the remote's own CLI produced, and that CLI may predate these fields. Without a default, adding one here refuses every remote running an older build, reported as "the remote CLI is probably a different version" -- true, and useless, since the older CLI is the one that cannot be changed. Fix CI: managed_runtime_roots resolved the data directory through crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default) instead of AppPaths::discover, so it disagreed with the rest of the CLI and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates the data directory -- which is exactly what every GPU e2e lane sets for isolation. That is why the new GPU scenario (examine-finds-the-managed-runtimes-own-compilation-library) failed identically on every GPU family; it now resolves through AppPaths::discover like everything else in the CLI. Also fixes a needless-collect clippy lint in the e2e step, and tags that scenario @requires-os:linux: the search looks only for libamd_comgr by its ELF soname, which native Windows does not ship. Rebasing onto main picked up two unrelated scenarios that claimed the same numbers in turn -- examine-16 (EAI-8950, ROCm#444), then examine-17 (EAI-8449, ROCm#434) -- so this commit's two new scenarios are renumbered to examine-18 and examine-19 to keep every scenario number in the file unique. Review: examine-18 (the host-independent code-object-manager scenario) was untagged and its empty-result assertion matched `Examination::default()` alone, so it would pass with `probe_comgr` deleted outright -- on native Windows, where the probe never runs, that is not a hypothetical. Tagged @requires-os:linux and the assertion now also requires the "no libamd_comgr found" note that only a probe which actually ran and came up empty pushes. Verified the new assertion fails by temporarily suppressing that note and watching the scenario go red, then restored it. Review: four more issues. (1) comgr_selected and the multi-copy note named the ld.so.cache system copy as "would load" whenever a managed wheel runtime was active but this process's own LD_LIBRARY_PATH was empty -- wrong, because rocm serve/rocm chat prepend the active runtime's own library directories onto LD_LIBRARY_PATH before launching the engine, so the wheel copy is what a served process actually loads. probe_comgr now takes the active runtime's interpreter (the same FrameworkInterpreter already threaded through probe_framework) and searches its library_paths first, ahead of the ambient environment; pinned by the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The site_packages: None fallback in collect_managed_runtime_library_paths guessed a venv layout (root/lib/<python>/site-packages) that the installer never produces -- root is one of the runtime's own _rocm_sdk_* package directories, not a venv root, so the guess always found nothing. It now reads root.parent() instead, which is the site-packages directory the guessed layout could never reach. The test fixture (wheel_runtime_on_disk) was built from the same wrong venv shape and so could not have caught this; rebuilt it from the real installer layout (ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed a_wheel_runtime_with_no_registry_record_is_still_found fails against the old fallback before passing against the fix. (3) probe_comgr, comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the selection, and the multi-copy note could all break with the rest of the suite still green. Split the ordering/selection logic into a pure find_comgr_copies, the ldconfig -p line parsing into parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings (opt directory taken as a parameter rather than hardcoded, so a test can point it at a temp folder instead of the real /opt on the machine running the suite); each is unit-tested directly, and each new test was confirmed to fail against a deliberately broken version of the function it covers before being restored. (4) This message previously claimed a HIP copy list and a single shared description of the loader path launched processes use; neither matches this diff. Corrected above. Review: a second round, one automated and one human, reaching the same place by different routes. The human reviewer confirmed, independently, that the site_packages: None fallback added in the prior round could not actually fire: ROCM_SDK_PROBE_SCRIPT has recorded site_packages unconditionally, outside its try, since the probe's initial version, so every manifest that parses at all carries Some there. The test built to cover it could not have caught that either way, since it called the function directly with None rather than through a real candidate. Rather than keep a reachable- looking branch and a comment describing behaviour the code does not have, the None arm is now a no-op (root's own directories, already collected unconditionally above, are what a hand-constructed None caller still gets) and the test is removed. The neighbouring root-format test is kept and re-targeted: it already called the same function directly with None, so it is what actually exercises that arm's contract, not the removed test. The automated reviewer caught a real regression the first round introduced: three new unit tests feed a path into the `:`-split LD_LIBRARY_PATH parser without a unix cfg gate, same as the pattern the file already uses for that reason, and that broke the required windows-build-and-test lane. Gated, along with the helper they share (which would otherwise be unused, and clippy-denied, once its only callers are gated). It also found that the active-runtime-priority fix regresses examine-finds-the-managed-runtimes-own-compilation-library on a healthy host: once a managed runtime is active, its directory is found first and labelled active-runtime, and the dedup this search has always done on resolved path then drops the later managed-runtime hit for the identical file -- the scenario's step asserted on the label that fix makes unreachable. The step now accepts either label as proof the CLI's own installed copy was found, and a new unit test (an_active_runtime_directory_that_is_also_a_managed_root_keeps_one_label) pins that an overlapping directory keeps exactly one label rather than being reported twice; confirmed to fail when the active-runtime search is moved after the others, then restored. Review: a third round, three issues. (1) comgr_matches_runtime was declared, defaulted, and listed in the expected-keys test, but nothing in this diff computes it -- comparing comgr's copies against the HIP runtime's is still deferred, same as the commit message already said. A reader of the JSON contract alone had no way to know the field was permanently unset rather than merely not yet exercised by a test. Dropped from this commit entirely (the struct field, its Default entry, its expected-keys assertion); `serde(default)` makes re-adding it free once something actually computes it. (2) comgr_selected named the first entry in comgr_paths as "would load" regardless of which tier found it, so a machine carrying only a leftover /opt/rocm-* install or an inactive managed runtime -- never a copy on active-runtime, ld-library-path, or loader-cache -- was told that stale copy would load, when the loader does not search /opt or a managed runtime's directory on its own. find_comgr_copies now selects only from those three loader-consulted tiers (LOADER_PATH_SOURCES), leaving comgr_selected None and rewording the note when every hit is rocm-install/managed-runtime-only; pinned by a_search_dir_only_hit_is_not_reported_as_the_one_that_would_load. The e2e step (assert_comgr_selection_is_stated) now asserts the same tier rule against the real JSON: selected absent plus only non-loader-tier copies is the pass case, not a shape to tolerate. (3) comgr_search_dirs and managed_runtime_roots had no unit tests of their own, and swapping the loader-cache and search-dir loops inside find_comgr_copies left every then-existing test green -- nothing pinned their relative order. Split comgr_search_dirs into a pure comgr_search_dirs_in (rocm_path, the /opt directory, and the resolved managed runtimes all taken as parameters) and added comgr_search_dirs_labels_each_tier, covering all three source labels against temp directories. Added loader_cache_outranks_search_dirs, built so an ld-library-path hit cannot mask the order under test. Confirmed the claim directly: swapped the two loops, reran the suite -- exactly loader_cache_outranks_search_dirs failed, the other 65 tests (including ld_library_path_outranks_loader_cache_and_search_dirs) stayed green -- then reverted the swap and reran clean before committing. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com> * feat(diagnose): report a code object manager that is not the runtime's own (ROCm#385) Builds on the copy search: with every copy of libamd_comgr known, the catalog can say when the one that loads belongs to a different installation than the HIP runtime that loads, and device code compilation therefore fails with an error naming neither. The rule is symmetric, and that is what makes it safe. The design note for this entry proposed a special case -- "when the active runtime is the managed runtime, the matching copy is the wheel copy" -- without which it "fires on every healthy CLI installation". That is a patch over a rule stated asymmetrically. Asking one question of both libraries instead, does the libamd_comgr that would load come from the same installation as the libamdhip64 that would load, makes every case fall out of the rule: a healthy managed install takes both from the managed runtime, so nothing differs and nothing is reported. A control that has to be written as an exception is a rule that has not been stated correctly yet. That turns on attributing a copy to its installation correctly. A managed runtime spreads its libraries across separate _rocm_sdk_* packages that sit beside each other under one site-packages, not nested under the runtime's own root at all, so ownership is decided by looking a found file's directory up against the same directories install_library_dirs already recorded for each root -- never by walking up from the file, which would call each package its own install and fire on the most common install we ship. That is the property the mutation check targets. Two further guards on firing: an unattributed copy is a gap in what the search knows rather than a finding about the machine, and the runtime must actually ship a copy of its own, or there is nothing to point the user at. The entry is print-only and ranks neither remedy. Removing a stack and reordering the search path can each break a working Python environment, and which is right depends on which stack the user means to keep. The finding says out loud that it describes the environment outside the managed runtimes. `rocm serve` puts a managed runtime's libraries first on purpose and gets a different, correct answer, and a report that did not name the environment it examined would read as a claim about one it never looked at. The rule above is now stated exactly once, as `comgr_matches_runtime` in examine.rs, and both this entry and `rocm examine --json`'s field of the same name call it rather than each restating the conditions -- so the two surfaces cannot disagree about one machine, and a regression test pins that. Fix option (b)'s advice matched an earlier, asymmetric statement of the rule; it now says to prefer the wheel's own copy of both libraries, which is what the symmetric rule actually asks the user to do. The machine-readable inspection now carries the same `install_root` attribution for the HIP runtime side that it already carried for the code object manager side, since the conflict question is about both libraries and a reader could not previously check either the attribution or the selection for one of them. Known roots are deduplicated across the whole list, not only where repeats happen to land next to each other. `dedup_by` drops consecutive duplicates, and these roots arrive from three independent sources -- the active install, a sorted /opt scan, and the managed runtimes -- so the active install colliding with a sibling was caught or missed depending on where the path sorted. A root recorded twice can make a machine holding one stack read as a machine holding two, which is the false conflict this entry exists not to report. hip_paths and hip_selected are `serde(default)` for the same reason the comgr fields are: an examination read back over the remote path may have been produced by an older CLI that never wrote them. Once a managed runtime is found, its root resolves to its `_rocm_sdk_devel` package directory, and the search used to guess a venv layout (`root/lib/<python>/site-packages`) to find its sibling packages; that guess does not hold for a real install, so the search now reads back the library directories the SDK probe already recorded (`library_paths`) instead of re-deriving them. This generalises the walker introduced in ROCm#384 (`find_library_copies`/`install_library_dirs`) so it serves the HIP side too, which is why `managed_runtime_roots` now returns `library_paths` plural rather than a single `site_packages` guess. The AppPaths::discover data-dir fix and the @requires-os:linux platform tags this commit originally carried for examine-17/18 belong with the scenario they correct, which ROCm#384 introduced -- moved there so the parent doesn't depend on the child to pass its own CI, and this commit now only rebases on top of that fix rather than restating it. Sweeping the same gap once it was visible: examine-17 (the host-independent code-object-manager scenario, renumbered by ROCm#384's rebase) asserts `hip_paths`/`hip_selected` the same way it asserts `comgr_paths`/`comgr_selected`, and the empty branch had the identical problem ROCm#384's review flagged for the comgr side -- `hip_paths: []` / `hip_selected: null` also hold by nothing more than `Examination`'s own defaults, so the assertion could not tell "probed, found none" from "never probed". `probe_comgr` now pushes a "no libamdhip64 found" note the same way it already did for libamd_comgr, and the HIP assertion now requires it in the empty branch. Verified by temporarily suppressing the note and watching the scenario go red, then restoring it. Review: the attribution half still did not survive a realistic managed runtime. `known_install_roots` registers a managed runtime by `candidate.root_path` alone -- `_rocm_sdk_devel` when the `devel` extra is installed -- and `libamd_comgr`/`libamdhip64` live in the sibling `_rocm_sdk_core` package, which sits *beside* `root_path`, not under it. A string-prefix match against the bare root therefore attributed nothing on exactly this shape: both libraries came back `install_root: ""`, `comgr_matches_runtime` hit its empty-root guard and returned `None` instead of `true`, and `check_18_comgr_conflict` could not fire for that runtime at all. The two unit tests meant to catch this (`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime`, `a_healthy_managed_installation_raises_no_report`) built their fixtures from a venv root (`/data/runtimes/therock/default`, `.../lib/python3.12/site-packages/_rocm_sdk_core/...`) that production never produces, so they passed regardless. Fixed at the root: `install_library_dirs` already walks, for every known root, exactly the directories that root's copies live in (that is the discovery-side fix this commit already made). `find_library_copies` now builds a directory -> owning-root map from that same walk and attributes each found file by looking its directory up in it, rather than by string-prefix match against the bare root. A sibling package's directory is simply one more entry in the map, keyed to the runtime that recorded it, so it attributes correctly without being nested under anything. `owning_install_root` and `record_library_copy` take that map instead of the bare root list; `the_nearest_enclosing_installation_claims_a_library` is removed because the "longest prefix wins" ambiguity it guarded against cannot arise under exact directory lookup. Both fixture tests above were rebuilt from the real installer shape (sibling `_rocm_sdk_*` packages under one `site-packages`, matching `apps/rocm/src/therock.rs`'s `ROCM_SDK_PROBE_SCRIPT`), and `a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` was confirmed to fail against the old string-prefix attribution before passing against the fix. `find_library_copies` also gained the `active_runtime_dirs` parameter runtime's own library directories are now searched first, ahead of the ambient `LD_LIBRARY_PATH`, for both comgr and HIP, since `rocm serve` prepends them for both libraries equally. The loader cache and the directory walk are now gathered once in `probe_comgr` and passed to both the comgr and HIP searches, rather than each search reading `LD_LIBRARY_PATH`, spawning `ldconfig -p`, and walking every managed runtime's directories again on every invocation -- neither depends on which library is being searched for, so the duplication cost an extra subprocess and directory walk on every `rocm examine`/`rocm diagnose` for no reason. `multi_copy_note` is split out as its own pure function so the "N copies ... would load" note stays independently testable now that `find_library_copies` only returns copies. Two doc-comment splices, both from a missing blank `///` line between two unrelated items: `check_18_comgr_conflict`'s doc paragraph had absorbed check_17's above it, leaving check_17 undocumented; the same shape left `known_install_roots` undocumented under a paragraph describing `dedup_roots_keeping_first`. Both are split back onto the function they actually describe. `docs/wsl.md` said `fix-19-shm-too-small` was the only non-WSL entry that applies on WSL; registering `check_18_comgr_conflict` for `["linux", "wsl"]` made that false the moment this entry existed. Updated to name both. The "Neither option is recommended..." sentence was stated independently in `fix.rs`'s static recipe and `diagnose.rs`'s dynamically-generated one, in slightly different words -- already drifted, and only the `fix.rs` copy was pinned by an e2e assertion, so the `diagnose.rs` copy could drift further with nothing to catch it. Both now read `fix::COMGR_CONFLICT_NEITHER_OPTION_RECOMMENDED`, so they cannot disagree. Not changed: `fix.rs`'s option (b) wording and `comgr_matches_runtime`'s parity with `check_18_comgr_conflict` were flagged in an earlier review round and were already correct by the time this round started -- verified against the current code rather than re-fixed. Rebase onto ROCm#384 and a second symmetry gap: ROCm#384's own review fixed `comgr_selected` picking `copies.first()` regardless of tier -- a `rocm-install`/`managed-runtime`-only hit used to be reported as "the one that would load" purely because it sorted first, when neither source is one the loader consults on its own. Rebasing that fix under this entry's HIP-runtime work surfaced a second instance of the same asymmetry a human reviewer caught directly on this branch: `probe_comgr`'s doc comment claimed comgr and HIP were searched and reported identically, but only comgr ever got a `multi_copy_note` call; the HIP branch only checked `hip.is_empty()` and pushed nothing for the "found, but only in a directory the loader does not consult" case the comgr side had just been fixed to report. Fixed by stating the tier rule once and applying it to both prefixes rather than fixing it twice: `select_loader_copy` replaces `copies.first()` with a search restricted to `LOADER_PATH_SOURCES` (`active-runtime`, `ld-library-path`, `loader-cache` -- the tiers the loader itself consults), and `copies_note` replaces `multi_copy_note` with the three-way note (no copies / more than one with a winner / one-or-more with none selected) that a `rocm-install`-only hit now needs. `probe_comgr` calls both exactly once for comgr and once for HIP, so there is one call site per library rather than one function's logic duplicated by hand into the other's branch, and the doc comment now says what the code does instead of what it used to do before ROCm#384's fix landed. Pinned by `a_search_dir_only_hit_is_not_selected_for_either_prefix`, which runs the same assertions for `COMGR_LIB_PREFIX` and `HIP_RUNTIME_LIB_PREFIX` in one test rather than two, so the rule is proven to apply to both rather than merely written to apply to both. The existing `loader_cache_outranks_search_dirs` and `the_active_runtimes_own_copy_outranks_the_ambient_environment`/ `ld_library_path_outranks_loader_cache_and_search_dirs` tests were carried through the rebase onto `find_library_copies`/ `select_loader_copy` rather than dropped, since they pin mutation-tested tier-ordering bugs ROCm#384 already fixed once. `comgr_search_dirs_labels_each_tier`, which tested the now-deleted `comgr_search_dirs_in`, is removed as redundant: `install_library_dirs`'s own tier-labelling is already covered by `a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` and `a_library_no_installation_claims_is_attributed_to_none`. Verified: `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo clippy -p e2e-cucumber --test e2e -- -D warnings`, and `cargo test --workspace --all-targets` all pass (one `xtask::workflow_contract` test flaked under full-suite parallel load on a timing assertion against a real `rocm-smi` subprocess call and passed cleanly in isolation and on a full-suite rerun; unrelated to this change). Cross-compiled the whole workspace, tests included, for `x86_64-pc-windows-gnu` under `-D warnings` to confirm no code is left unix-only-reachable and therefore dead on Windows. `cargo xtask e2e -- -n skill` passes all 6 scenarios, since this round also touches `skills/rocm-doctor/reference.md`. Second review round, two independent reviewers filing the same four findings (verified rather than merely accepted): fix-18's evidence line claimed to describe "the environment as it stands outside the CLI's managed runtimes" unconditionally, even when a selection actually came from the active-runtime tier -- a served child's search order, not a plain shell's. That also means the `export LD_LIBRARY_PATH=...` remedy can be inert for exactly that case, since `lemonade_process_environment_vars` prepends the runtime's own directories ahead of whatever the remedy exports. Rewording the evidence line is the easy half; the detection logic that decides which sentence to print cannot move to plain-shell-only without reintroducing the false positives/negatives `comgr_matches_runtime` and `a_healthy_managed_installation_raises_no_report` were written to rule out, since that guarantee depends on both selections already being active-runtime-tier-aware. So `comgr_matches_runtime` is untouched; `check_18_comgr_conflict` now checks whether either selected copy's source is `active-runtime` and, when it is, swaps in an honest sentence plus a new remedy note stating the export may not reach a served child and pointing at reinstalling/repairing that runtime instead. Pinned by `an_active_runtime_s_own_hip_without_its_own_comgr_notes_the_scope_limit`, mutation-tested by forcing the branch condition false and watching the new test fail on the stale sentence, then reverting. `assert_hip_selection_is_stated`, the e2e HIP twin of the comgr selection assertion, had only two branches, so "copies found, but none on a tier the loader consults" -- `select_loader_copy` returning `None` on purpose, already pinned for `libamdhip64` by `a_search_dir_only_hit_is_not_selected_for_either_prefix` -- fell into the `else` arm and asserted against a selected copy that does not exist. Added the missing third branch, mirroring `assert_comgr_selection_is_stated` exactly, including its `LOADER_PATH_SOURCES` check on the selected copy's source, which the HIP step's final arm was also missing. Hoisted the shared `LOADER_PATH_SOURCES` const the two steps now both use so they cannot drift apart again. `skills/rocm-doctor/reference.md`'s "Closed catalog (24 failure modes)" heading and `SKILL.md`'s "the other 20 are print-only" line had both gone stale against the table's 25 rows and 4 auto-applicable fixes (`exactly_the_four_known_fixes_are_auto`); fixed to 25 and 21. `rocm_doctor_skill.feature` deliberately leaves both counts unparsed against the live CLI, and that scope decision stands -- the CLI is not wired to report "how many rows exist" as a fact an agent reads. What it does not rule out is checking the two numbers against each other, so `tests/e2e-cucumber/tests/skill_reference.rs` (plain `cargo test`, no binary needed) gained `free_standing_catalog_counts_match_the_table`, which parses both numbers back out of the two docs and asserts them against the table's own row and auto-fix counts. Mutation-tested by reverting each number one at a time and confirming the new test goes red, then restoring it. Open question from the first round, never answered: does `install_library_dirs` calling `collect_sdk_library_paths` unconditionally widen what a bare-metal `/opt/rocm` root probes in a way that matters? Traced it rather than taking the earlier answer on faith: every directory `install_library_dirs` adds for a root carries that root's own source (`rocm-install` or `managed-runtime`, never a loader-consulted tier), and `find_library_copies` only records a copy under that source when no loader-consulted tier (`active-runtime`/`ld-library-path`/`loader-cache`) already found the same path first -- so a wider probe can only add more `rocm-install`-sourced evidence, never promote a path onto a tier `select_loader_copy` would pick. The one place a wider probe does change an answer is `comgr_matches_runtime`'s "does the runtime ship its own copy" check, and there a new hit can only turn a false conflict report into `None` (not enough evidence), which is strictly more correct, not a new risk. The reviewer's reasoning holds; recorded here since the first round left it unanswered. Re-verified after this round's changes: `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo clippy -p e2e-cucumber --test e2e -- -D warnings`, `cargo test --workspace --all-targets` (251 passed, 0 failed across 33 binaries), and `RUSTFLAGS="-D warnings" cargo check --workspace --all-targets --target x86_64-pc-windows-gnu` all pass clean. `cargo xtask e2e -- -n skill` passes all 6 scenarios again; `cargo xtask e2e -- -n examine-18` passes its 5 steps, confirming the new HIP e2e branch does not break the existing scenario. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com> --------- Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
What
#384 made every copy of
libamd_comgrvisible. This adds the diagnosis: the catalog now reports when the copy that would load belongs to a different installation than the HIP runtime that would load — the state in which device code compilation fails with an error naming neither the library nor the second copy.Why the rule is symmetric
The design note for this entry proposed a special case: "when the active HIP runtime is the managed runtime, the matching copy is the wheel copy. Without this rule the entry fires on every healthy CLI installation."
That is a patch over a rule stated asymmetrically. Asking one question of both libraries instead — does the
libamd_comgrthat would load come from the same installation as thelibamdhip64that would load? — makes every case fall out of the rule rather than being legislated:/opt/rocm/opt/rocmA control that has to be written as an exception is a rule that has not been stated correctly yet.
The detail it all turns on
A managed runtime spreads its libraries across separate
_rocm_sdk_*packages that sit beside each other under onesite-packages, not nested under the runtime's own root (_rocm_sdk_devel, when thedevelextra is installed) at all. Ownership is therefore decided by looking a found file's directory up against the same directoriesinstall_library_dirsalready walks for each root — never a string-prefix match against the bare root, which attributed nothing on exactly this shape and never by walking up from the file either, which would call each package its own installation and fire on the most common install we ship.That is not a theoretical concern — an earlier version of this PR used a string-prefix match, which left every managed runtime installed with the
develextra unattributed (comgr_matches_runtime: nullinstead oftrueon a healthy install), and the two unit tests meant to catch it were built from a venv-shaped fixture production never produces, so they passed anyway. See "Review remediation" below.Two further guards on firing
It advises and ranks nothing
Print-only. Removing a stack and reordering the search path can each break a working Python environment, and which is right depends on which stack the user means to keep — a question only they can answer. The finding also states that it describes the environment outside the managed runtimes:
rocm serveputs a managed runtime's libraries first on purpose and gets a different, correct answer, and a report that did not name the environment it examined would read as a claim about one it never looked at.Verification
cargo fmt --all -- --check,cargo clippy --workspace --all-targets -- -D warnings, andcargo clippy -p e2e-cucumber --test e2e -- -D warnings(theharness = falsetarget--all-targetsskips) — all cleancargo test --workspace --all-targets— clean, run twice after the review round below to rule out the one unrelated parallel-execution flake (restarting_a_keyless_public_service_names_the_public_bind_not_the_key_flag, a filesystem path race under full-suite parallelism, confirmed to pass in isolation) as masking a real failureEvery new assertion was verified to fail, not just to pass, including the ones added in the review round below — each was confirmed red against a deliberately broken version of the thing it covers, then restored.
CI
The scenario this PR's own GPU coverage depends on,
examine-finds-the-managed-runtimes-own-compilation-library, isexamine-19in the currentexamine.feature(renumbered again by #384's own rebase onto a newermain; earlier revisions of this PR body called itexamine-17, which is now a different, unrelated scenario). It was introduced in #384, not here — along with theAppPaths::discoverdata-dir fix and the@requires-os:linuxtag that make it pass on every GPU lane including native Windows's absence. Nothing in this PR's own commit touches that scenario or its fixes; it only rebases on top of them.Review remediation
A human review found a real gap in this PR's own commit, beyond the mechanical renumbering above: see "The detail it all turns on" for the attribution bug on managed runtimes installed with the
develextra, and its fix. Also fixed in this round, all in this PR's own commit:///line left one function's docs attached to the next item instead):check_18_comgr_conflicthad absorbedcheck_17's doc paragraph, leavingcheck_17undocumented;known_install_rootswas undocumented the same way, underdedup_roots_keeping_first's docs.docs/wsl.mdsaidfix-19-shm-too-smallwas the only non-WSL entry that applies on WSL; registeringcheck_18_comgr_conflictfor["linux", "wsl"]made that stale. Updated to name both.fix.rs's static recipe anddiagnose.rs's dynamically-generated one, and only thefix.rscopy was pinned by an e2e assertion. Both now read a sharedfix::COMGR_CONFLICT_NEITHER_OPTION_RECOMMENDEDconstant, so they cannot drift apart.find_library_copiesalso picked up the active-managed-runtime search-priority fix feat(examine): report every code object manager library, not just the first #384's own review round added to the comgr-only search (generalized here to cover HIP too), and the loader cache / directory walk are now gathered once perrocm examine/rocm diagnoseinvocation and shared between the comgr and HIP searches rather than duplicated.fix.rs's option (b) wording, andcomgr_matches_runtime's parity withcheck_18_comgr_conflict) were checked against the current code and found already correct — not re-fixed, just verified.Full detail and verification is in the commit message's second
Review:paragraph.Review remediation, round two
A second round of review (two independent reviewers, filing the same four findings) found one real logic gap and one unanswered question from round one; the other two were doc/test hygiene.
active-runtimetier — a served child's search order, not a plain shell's — in which case theexport LD_LIBRARY_PATH=...remedy a few lines below is not guaranteed to reach whatrocm serve/rocm chatactually load. Rewording the sentence alone would have left the remedy inert for exactly that case.check_18_comgr_conflictnow checks whether either selected copy's source isactive-runtimeand, when it is, swaps in an honest sentence ("an active managed runtime decided part of this...") plus an added fix note that the export may not reach a served child, pointing at reinstalling/repairing that runtime instead.comgr_matches_runtimeitself is untouched — only which sentence/note gets emitted changes. Pinned byan_active_runtime_s_own_hip_without_its_own_comgr_notes_the_scope_limit.assert_hip_selection_is_stated's missing third branch.select_loader_copyreturningNonefor HIP specifically (copies found, none on a loader-consulted tier) is contract, already pinned bya_search_dir_only_hit_is_not_selected_for_either_prefix, but the HIP e2e step only had two branches and asserted against a selection that doesn't exist in that state. Fixed by unifying the comgr/HIP note-building logic (select_loader_copyplus a newcopies_note) rather than copy-pasting the comgr branch over, since the oldmulti_copy_note/.first()pick had its own tier-blind asymmetry the copy-paste would have preserved. Both e2e steps now share oneLOADER_PATH_SOURCESconst and the same three branches.reference.md's "(24 failure modes)" heading andSKILL.md's "the other 20 are print-only" line, against a table that had grown to 25 rows / 21 non-auto-applicable. Fixed to 25/21, andtests/e2e-cucumber/tests/skill_reference.rsgainedfree_standing_catalog_counts_match_the_table, which parses both numbers back out of the docs and checks them against the table's own counts — narrower than, and not a reversal of,rocm_doctor_skill.feature's documented decision to leave these two numbers unparsed against the live CLI.Re-verified after this round:
cargo fmt --all -- --check,cargo clippy --workspace --all-targets -- -D warnings,cargo clippy -p e2e-cucumber --test e2e -- -D warnings,cargo test --workspace --all-targets(251 passed, 0 failed across 33 binaries),RUSTFLAGS="-D warnings" cargo check --workspace --all-targets --target x86_64-pc-windows-gnu,cargo xtask e2e -- -n skill(6/6 scenarios),cargo xtask e2e -- -n examine-18(5/5 steps, confirming the new HIP e2e branch doesn't regress the existing scenario). Every new or changed assertion was mutation-tested: broken deliberately, confirmed red, reverted, confirmed green.Open question: does widening the bare-metal scan matter?
install_library_dirscallscollect_sdk_library_pathsunconditionally for every known root, widening what a plain/opt/rocminstall probes compared to the narrowerrocm-install-only scan this replaced. Traced the full chain rather than asserting an answer: every directory this adds is tagged with that root's own source (rocm-install/managed-runtime), andfind_library_copiesonly records a copy under a given source if no loader-consulted tier (active-runtime/ld-library-path/loader-cache) already found the same path first — so a wider probe can add only morerocm-install-sourced evidence, never promote a path onto a tierselect_loader_copywould pick, and so it cannot changecomgr_selected/hip_selectedor what "would load" means. The one place the wider evidence list matters iscomgr_matches_runtime's "does the runtime ship its own copy" check, and there a new hit can only turn a false-conflictSome(false)intoNone(not enough evidence) — strictly more conservative, not a new risk. Confirmed, not just repeated: intentional and safe.Coverage boundary
The suite cannot install a second ROCm stack, so e2e cannot provoke a real conflict. It pins the half that matters if the entry ever stops being advisory — asking for the fix changes nothing and recommends neither option. The detection rule is proven by unit tests that construct the machine state directly, including the two cases that must stay silent.
Risk
Medium, concentrated in one place: a false report on a healthy managed installation. That is the risk the ticket names, and the symmetric rule plus root-based attribution are the two things standing against it, each with a test that fails when broken.
Follow-up
Overlaps #382, which replaces
auto_applicablewith a per-platform class. This branch is off #384, so the new recipe uses the current bool; whichever lands second rebases. If #382 lands first this entry becomesPRINT-ONLYin the new class directly.