Skip to content

feat(diagnose): report a code object manager that is not the runtime's own - #385

Merged
volen-silo merged 1 commit into
feat/report-comgr-copiesfrom
feat/diagnose-comgr-conflict
Oct 5, 2026
Merged

volen-silo merged 1 commit into
feat/report-comgr-copiesfrom
feat/diagnose-comgr-conflict

Conversation

@volen-silo

@volen-silo volen-silo commented Sep 11, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #384 — based on feat/report-comgr-copies, so this diff is just the one commit. Will be rebased onto main once #384 lands.

What

#384 made every copy of libamd_comgr visible. This adds the diagnosis: the catalog now reports when the copy that would load belongs to a different installation than the HIP runtime that would load — the state in which device code compilation fails with an error naming neither the library nor the second copy.

Why the rule is symmetric

The design note for this entry proposed a special case: "when the active HIP runtime is the managed runtime, the matching copy is the wheel copy. Without this rule the entry fires on every healthy CLI installation."

That is a patch over a rule stated asymmetrically. Asking one question of both libraries instead — does the libamd_comgr that would load come from the same installation as the libamdhip64 that would load? — makes every case fall out of the rule rather than being legislated:

Situation Owning installs Fires?
Healthy system-only install both /opt/rocm No
Healthy managed install both the managed runtime No — no special case needed
Wheel lib dir on the search path, runtime from /opt/rocm differ Yes
Two copies, the right one wins same No, and both stay listed
One copy, or none nothing to differ No

A control that has to be written as an exception is a rule that has not been stated correctly yet.

The detail it all turns on

A managed runtime spreads its libraries across separate _rocm_sdk_* packages that sit beside each other under one site-packages, not nested under the runtime's own root (_rocm_sdk_devel, when the devel extra is installed) at all. Ownership is therefore decided by looking a found file's directory up against the same directories install_library_dirs already walks for each root — never a string-prefix match against the bare root, which attributed nothing on exactly this shape and never by walking up from the file either, which would call each package its own installation and fire on the most common install we ship.

That is not a theoretical concern — an earlier version of this PR used a string-prefix match, which left every managed runtime installed with the devel extra unattributed (comgr_matches_runtime: null instead of true on a healthy install), and the two unit tests meant to catch it were built from a venv-shaped fixture production never produces, so they passed anyway. See "Review remediation" below.

Two further guards on firing

  • An unattributed copy is not a mismatch. It means the search found a library no known installation claims — a gap in what the probe knows, not a finding about the user's machine.
  • The runtime must actually ship a copy of its own, or there is nothing to switch to and the advice would be empty.

It advises and ranks nothing

Print-only. Removing a stack and reordering the search path can each break a working Python environment, and which is right depends on which stack the user means to keep — a question only they can answer. The finding also states that it describes the environment outside the managed runtimes: rocm serve puts a managed runtime's libraries first on purpose and gets a different, correct answer, and a report that did not name the environment it examined would read as a claim about one it never looked at.

Verification

  • cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings, and cargo clippy -p e2e-cucumber --test e2e -- -D warnings (the harness = false target --all-targets skips) — all clean
  • cargo test --workspace --all-targets — clean, run twice after the review round below to rule out the one unrelated parallel-execution flake (restarting_a_keyless_public_service_names_the_public_bind_not_the_key_flag, a filesystem path race under full-suite parallelism, confirmed to pass in isolation) as masking a real failure
  • GPU e2e lanes run in CI, not locally; see "CI" below

Every new assertion was verified to fail, not just to pass, including the ones added in the review round below — each was confirmed red against a deliberately broken version of the thing it covers, then restored.

CI

The scenario this PR's own GPU coverage depends on, examine-finds-the-managed-runtimes-own-compilation-library, is examine-19 in the current examine.feature (renumbered again by #384's own rebase onto a newer main; earlier revisions of this PR body called it examine-17, which is now a different, unrelated scenario). It was introduced in #384, not here — along with the AppPaths::discover data-dir fix and the @requires-os:linux tag that make it pass on every GPU lane including native Windows's absence. Nothing in this PR's own commit touches that scenario or its fixes; it only rebases on top of them.

Review remediation

A human review found a real gap in this PR's own commit, beyond the mechanical renumbering above: see "The detail it all turns on" for the attribution bug on managed runtimes installed with the devel extra, and its fix. Also fixed in this round, all in this PR's own commit:

  • Two doc-comment splices (a missing blank /// line left one function's docs attached to the next item instead): check_18_comgr_conflict had absorbed check_17's doc paragraph, leaving check_17 undocumented; known_install_roots was undocumented the same way, under dedup_roots_keeping_first's docs.
  • docs/wsl.md said fix-19-shm-too-small was the only non-WSL entry that applies on WSL; registering check_18_comgr_conflict for ["linux", "wsl"] made that stale. Updated to name both.
  • The "Neither option is recommended..." sentence was stated independently (and already slightly differently worded) in fix.rs's static recipe and diagnose.rs's dynamically-generated one, and only the fix.rs copy was pinned by an e2e assertion. Both now read a shared fix::COMGR_CONFLICT_NEITHER_OPTION_RECOMMENDED constant, so they cannot drift apart.
  • find_library_copies also picked up the active-managed-runtime search-priority fix feat(examine): report every code object manager library, not just the first #384's own review round added to the comgr-only search (generalized here to cover HIP too), and the loader cache / directory walk are now gathered once per rocm examine/rocm diagnose invocation and shared between the comgr and HIP searches rather than duplicated.
  • Two earlier review comments (fix.rs's option (b) wording, and comgr_matches_runtime's parity with check_18_comgr_conflict) were checked against the current code and found already correct — not re-fixed, just verified.

Full detail and verification is in the commit message's second Review: paragraph.

Review remediation, round two

A second round of review (two independent reviewers, filing the same four findings) found one real logic gap and one unanswered question from round one; the other two were doc/test hygiene.

  • fix-18's evidence line was unconditionally wrong for one case. It asserted the finding "describes the environment as it stands outside the CLI's managed runtimes" even when a selection came from the active-runtime tier — a served child's search order, not a plain shell's — in which case the export LD_LIBRARY_PATH=... remedy a few lines below is not guaranteed to reach what rocm serve/rocm chat actually load. Rewording the sentence alone would have left the remedy inert for exactly that case. check_18_comgr_conflict now checks whether either selected copy's source is active-runtime and, when it is, swaps in an honest sentence ("an active managed runtime decided part of this...") plus an added fix note that the export may not reach a served child, pointing at reinstalling/repairing that runtime instead. comgr_matches_runtime itself is untouched — only which sentence/note gets emitted changes. Pinned by an_active_runtime_s_own_hip_without_its_own_comgr_notes_the_scope_limit.
  • assert_hip_selection_is_stated's missing third branch. select_loader_copy returning None for HIP specifically (copies found, none on a loader-consulted tier) is contract, already pinned by a_search_dir_only_hit_is_not_selected_for_either_prefix, but the HIP e2e step only had two branches and asserted against a selection that doesn't exist in that state. Fixed by unifying the comgr/HIP note-building logic (select_loader_copy plus a new copies_note) rather than copy-pasting the comgr branch over, since the old multi_copy_note/.first() pick had its own tier-blind asymmetry the copy-paste would have preserved. Both e2e steps now share one LOADER_PATH_SOURCES const and the same three branches.
  • Two free-standing prose counts went stale: reference.md's "(24 failure modes)" heading and SKILL.md's "the other 20 are print-only" line, against a table that had grown to 25 rows / 21 non-auto-applicable. Fixed to 25/21, and tests/e2e-cucumber/tests/skill_reference.rs gained free_standing_catalog_counts_match_the_table, which parses both numbers back out of the docs and checks them against the table's own counts — narrower than, and not a reversal of, rocm_doctor_skill.feature's documented decision to leave these two numbers unparsed against the live CLI.
  • The open question below, carried over unanswered from round one.

Re-verified after this round: cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings, cargo clippy -p e2e-cucumber --test e2e -- -D warnings, cargo test --workspace --all-targets (251 passed, 0 failed across 33 binaries), RUSTFLAGS="-D warnings" cargo check --workspace --all-targets --target x86_64-pc-windows-gnu, cargo xtask e2e -- -n skill (6/6 scenarios), cargo xtask e2e -- -n examine-18 (5/5 steps, confirming the new HIP e2e branch doesn't regress the existing scenario). Every new or changed assertion was mutation-tested: broken deliberately, confirmed red, reverted, confirmed green.

Open question: does widening the bare-metal scan matter?

install_library_dirs calls collect_sdk_library_paths unconditionally for every known root, widening what a plain /opt/rocm install probes compared to the narrower rocm-install-only scan this replaced. Traced the full chain rather than asserting an answer: every directory this adds is tagged with that root's own source (rocm-install/managed-runtime), and find_library_copies only records a copy under a given source if no loader-consulted tier (active-runtime/ld-library-path/loader-cache) already found the same path first — so a wider probe can add only more rocm-install-sourced evidence, never promote a path onto a tier select_loader_copy would pick, and so it cannot change comgr_selected/hip_selected or what "would load" means. The one place the wider evidence list matters is comgr_matches_runtime's "does the runtime ship its own copy" check, and there a new hit can only turn a false-conflict Some(false) into None (not enough evidence) — strictly more conservative, not a new risk. Confirmed, not just repeated: intentional and safe.

Coverage boundary

The suite cannot install a second ROCm stack, so e2e cannot provoke a real conflict. It pins the half that matters if the entry ever stops being advisory — asking for the fix changes nothing and recommends neither option. The detection rule is proven by unit tests that construct the machine state directly, including the two cases that must stay silent.

Risk

Medium, concentrated in one place: a false report on a healthy managed installation. That is the risk the ticket names, and the symmetric rule plus root-based attribution are the two things standing against it, each with a test that fails when broken.

Follow-up

Overlaps #382, which replaces auto_applicable with a per-platform class. This branch is off #384, so the new recipe uses the current bool; whichever lands second rebases. If #382 lands first this entry becomes PRINT-ONLY in the new class directly.

@volen-silo
volen-silo force-pushed the feat/report-comgr-copies branch from 7d0af16 to 9452104 Compare September 11, 2026 11:22
@volen-silo
volen-silo force-pushed the feat/diagnose-comgr-conflict branch from 87cbcac to a1968d9 Compare September 11, 2026 11:23
@volen-silo
volen-silo force-pushed the feat/report-comgr-copies branch from 9452104 to 4e5fe59 Compare September 11, 2026 11:35
@volen-silo
volen-silo force-pushed the feat/diagnose-comgr-conflict branch from a1968d9 to 22e9b4a Compare September 11, 2026 11:39
@volen-silo
volen-silo force-pushed the feat/report-comgr-copies branch from 4e5fe59 to 6f617c8 Compare September 15, 2026 13:18
@volen-silo
volen-silo force-pushed the feat/diagnose-comgr-conflict branch from 22e9b4a to 5c39eee Compare September 15, 2026 13:24
@volen-silo
volen-silo force-pushed the feat/report-comgr-copies branch from 6f617c8 to 09476f8 Compare September 28, 2026 07:53
@volen-silo
volen-silo force-pushed the feat/diagnose-comgr-conflict branch from 5c39eee to f1a1b9b Compare September 28, 2026 07:56
@volen-silo
volen-silo marked this pull request as ready for review September 28, 2026 10:41
@volen-silo
volen-silo requested a review from a team as a code owner September 28, 2026 10:41

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a solid feature (surfacing comgr/HIP runtime mismatches), but I found a few issues that affect correctness of the diagnosis output and the fix advice text, plus a couple of secondary concerns. Left inline comments on the ones anchored to this diff; one test-coverage gap below since it points at a file this PR doesn't touch.

Untested field, tests/e2e-cucumber/tests/e2e/examine_steps.rs (not modified by this PR): the e2e assertion for comgr_paths entries still only requires ["path", "real_path", "version", "source"] and wasn't extended to require the new install_root field this PR adds to the wire format, and no e2e step checks hip_paths/hip_selected either. Right now only in-process unit tests exercise install_root — the actual CLI --json output for the field this design turns on is unverified end-to-end.

Requesting changes mainly for the fix-18 advice/data mismatch (correctness bug in user-facing guidance) and the comgr_matches_runtime vs diagnose disagreement — both are logic errors, not style nits.

Comment thread crates/rocm-core/src/fix.rs Outdated
"# Then pick ONE of the following. They are alternatives, not steps.",
"# (a) Keep the system installation: remove or uninstall the wheel that",
"# supplies the second copy.",
"# (b) Keep the wheel: order the search path so the runtime's own copy",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Option (b) here says "Keep the wheel: order the search path so the runtime's own copy is found first", followed by an LD_LIBRARY_PATH command — but per this PR's own conflict definition (diagnose.rs's matching), "the runtime's own copy" is the copy belonging to the currently-active HIP runtime, which in the canonical wheel/system conflict is the system install, not the wheel. A user who wants to keep the wheel and follows this advice literally will export LD_LIBRARY_PATH to the system comgr directory instead — the opposite of the stated intent — and there's no obvious symptom afterward besides now picking up the system's libraries. Worth double-checking which path this recipe actually resolves to for option (b) and fixing the wording/command so it points at the wheel's own comgr, not the runtime's.

Comment thread crates/rocm-core/src/examine.rs Outdated
// `None` rather than `false` when either side is missing: "they disagree" and
// "there was nothing to compare" are different answers, and a caller that
// cannot tell them apart reports a conflict on a machine with no ROCm at all.
e.comgr_matches_runtime = match (comgr.first(), hip.first()) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documented --json field comgr_matches_runtime is computed with a weaker rule (comgr.install_root != hip.install_root, both non-empty) than check_18_comgr_conflict, which additionally requires the active runtime to ship a matching comgr copy of its own before firing. When the runtime ships no comgr copy of its own — the exact case this PR's own test a_runtime_with_no_copy_of_its_own_is_not_a_conflict covers — rocm examine --json will report comgr_matches_runtime: false while rocm diagnose reports nothing at all. Anyone consuming this field programmatically (or a user comparing the two commands) sees a contradiction. Suggest computing this field with the same logic check_18_comgr_conflict uses.

Comment thread crates/rocm-core/src/examine.rs Resolved
Comment thread crates/rocm-core/src/examine.rs Outdated
let mut copies: Vec<LibraryCopy> = Vec::new();
let mut seen: std::collections::BTreeSet<String> = std::collections::BTreeSet::new();

// Order is the whole point: this is the order the loader consults, so the

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

find_library_copies recomputes install_library_dirs(roots) and re-invokes paths_in_loader_cache (which spawns ldconfig -p) once per prefix, so both now run twice per rocm diagnose/rocm examine call — once for comgr, once for hip — even though neither depends on which prefix is being searched. That's an extra ldconfig -p subprocess plus a second full walk of every managed runtime's lib/lib64 → pythonX/site-packages → _rocm_sdk_* directories on every invocation of a command that's typically run repeatedly while debugging. Might be worth hoisting the roots/loader-cache computation out so it's shared between the two calls.

Comment thread crates/rocm-core/src/examine.rs Outdated
.into_iter()
.map(|root| (root, "managed-runtime")),
);
roots.dedup_by(|a, b| a.0 == b.0);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

known_install_roots dedups with Vec::dedup_by, which only removes adjacent duplicates. e.rocm_path is pushed first and the sorted /opt/* siblings are appended after, so if e.rocm_path equals one of those sorted entries but isn't adjacent to it in the final vector (e.g. e.rocm_path is /opt/rocm-6.4 and /opt also contains rocm and rocm-6.4, sorting to [rocm, rocm-6.4]), the roots vector ends up [rocm-6.4, rocm, rocm-6.4, ...] and the two rocm-6.4 entries survive the dedup. install_library_dirs/find_library_copies then scan that install's directories twice per probe — masked from the final output only because record_library_copy separately dedups by resolved real path downstream. Sorting the full vector (or using a HashSet) before dedup would close this.

@volen-silo
volen-silo force-pushed the feat/diagnose-comgr-conflict branch from f1a1b9b to c591f8a Compare September 29, 2026 11:56
@volen-silo
volen-silo force-pushed the feat/report-comgr-copies branch 2 times, most recently from d9b0a75 to f2bd0c9 Compare September 29, 2026 12:05
@volen-silo
volen-silo force-pushed the feat/diagnose-comgr-conflict branch from c591f8a to 2a77f96 Compare September 29, 2026 12:17
@volen-silo
volen-silo force-pushed the feat/report-comgr-copies branch 2 times, most recently from 6ea3a21 to ee48870 Compare September 29, 2026 13:23
@volen-silo
volen-silo force-pushed the feat/diagnose-comgr-conflict branch from 2a77f96 to c6cfd42 Compare September 29, 2026 13:33
@volen-silo
volen-silo force-pushed the feat/report-comgr-copies branch from ee48870 to a8749a3 Compare September 30, 2026 06:49
@volen-silo
volen-silo force-pushed the feat/diagnose-comgr-conflict branch 3 times, most recently from 991f5d8 to 166d9ed Compare September 30, 2026 14:33
@r0x0r r0x0r added the agent-hub-reviewing agent-hub review in progress label Oct 1, 2026

@r0x0r r0x0r left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What the change does. Teaches examine/diagnose to notice that the libamd_comgr copy the loader would pick belongs to a different installation than the libamdhip64 runtime, reports the HIP copies alongside the comgr ones, and adds fix-18-comgr-conflict as an advisory-only entry that states both remedies without ranking them.

Coverage. Full read of crates/rocm-core/src/{examine,diagnose,fix,lib}.rs and the four e2e-cucumber files at head, reviewed against the PR's own base (feat/report-comgr-copies), so #384's changes are excluded. Checked against AGENTS.md §3. Both findings from the earlier, now-dismissed review were re-tested specifically. No tests run.

Assessment: looks sound — no blocking findings survived verification.

The earlier review's two findings are addressed

  • fix-18 advice/data mismatch — fixed. The machine-specific Fix in diagnose.rs derives its LD_LIBRARY_PATH directory from matching.real_path, where matching is the comgr copy whose install_root equals the HIP runtime's, so "order the search path so the runtime's own copy is found first" now describes what the command actually does. The <directory of the wheel's own copy> placeholder that remains in fix.rs is the generic catalog recipe, which has no live paths to fill in — correct for its role.

  • comgr_matches_runtime vs diagnose disagreeing — fixed. probe_comgr and check_18_comgr_conflict now call the same examine::comgr_matches_runtime, and comgr_matches_runtime_never_disagrees_with_the_diagnosis pins that invariant across the catalog cases, including the one that used to diverge (a runtime shipping no comgr copy of its own). Making the disagreement structurally impossible, rather than fixing the two sites to agree, is the right shape of fix.

  • The e2e gap is closed too. assert_comgr_copies_reported now requires install_root:

    for field in ["path", "real_path", "version", "source", "install_root"] {

    and two new steps assert hip_paths and hip_selected, wired into examine-16.

AGENTS.md §3 is satisfied: diagnose-21 (@id:diagnose-fix-comgr-conflict-is-advisory-only) covers the user-observable half, and its comment explains honestly why the conflict itself cannot be provoked in the suite — the detection rule is left to unit tests that build machine state directly. That is the explanation §3 asks for rather than a silent omission.

Non-blocking

  • managed_runtime_roots() runs twice per find_library_copies. known_install_roots (examine.rs:2170) calls it to build roots, then install_library_dirs (:2200) calls it again to build managed_library_paths, and probe_comgr runs the pair twice (comgr, then HIP) — four registry walks per examination. It is a deterministic filesystem read, so this is cost rather than correctness, but threading the first result through would remove both the cost and any question about the two views agreeing.
  • comgr_version_from_file_name is now called for HIP paths too (record_library_copy). The soname parsing is correct for both; only the name still says comgr.
  • assert_hip_selection_is_stated passes trivially on the mock lane — with hip_paths == [] it only asserts hip_selected is null, so the list/selection agreement is never exercised off a GPU lane. It inherits this from the comgr sibling rather than introducing it.
  • assert_fix_is_advisory uses contains("print-only") || contains("will NOT run it"), so dropping either string still passes. assert_both_options_unranked covers the dangerous case, so this is slack rather than a hole.
  • Untested: the ordering contract in find_library_copies (LD_LIBRARY_PATH beats loader cache beats install dirs) is the whole point of the function and has no unit test; neither does the partial-version evidence branch in diagnose.rs where exactly one side has a version string.

Tradeoffs

  • comgr_matches_runtime returns None both for "no known installation claims this copy" and for "the runtime ships no comgr of its own". Serialized, a consumer cannot tell those apart from "not computed". The comments document the distinction; the wire format does not carry it.
  • The symmetric rule ("do both come from the same installation?") stays quiet on a machine holding only the mismatched comgr, because there would be no alternative to point at. Deliberate and documented, but it does mean a genuinely broken single-stack install reports nothing.

Positive signals

  • Driving the JSON field and the diagnostic from one function, then testing that they agree, removes a whole class of future drift.
  • Longest-prefix attribution in owning_install_root handles managed runtimes spread across _rocm_sdk_* subdirectories without special cases, and dedup_roots_keeping_first has a test for the non-adjacent duplicate that a naive consecutive dedup would miss.
  • auto_applicable: false, with both remedies stated and neither recommended, is the right posture for a fix where either choice can break a working Python environment.
  • #[serde(default)] on every new field keeps older consumers reading the new output.

One thing outside the diff

This PR's base is #384, which currently has changes requested and 6 failing checks, so #385 cannot land ahead of it regardless of its own state. Its own checks are green at this head apart from one still running.

The merge decision is yours; this review is posted as a comment and files no approval or change request.


🤖 by agent-hub on AMD AgentHub

@r0x0r r0x0r added agent-hub-reviewed agent-hub has reviewed this and removed agent-hub-reviewing agent-hub review in progress labels Oct 1, 2026
@volen-silo
volen-silo force-pushed the feat/diagnose-comgr-conflict branch from 166d9ed to 1837ea8 Compare October 1, 2026 10:05
@volen-silo
volen-silo force-pushed the feat/report-comgr-copies branch from a8749a3 to 51a20ff Compare October 1, 2026 13:17
@volen-silo
volen-silo force-pushed the feat/report-comgr-copies branch from 51a20ff to c5c15f7 Compare October 2, 2026 05:57
@volen-silo
volen-silo force-pushed the feat/diagnose-comgr-conflict branch from 4236280 to 19e0fda Compare October 2, 2026 06:23
@volen-silo
volen-silo force-pushed the feat/report-comgr-copies branch from c5c15f7 to a6d7af6 Compare October 2, 2026 13:10

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR and found one real gap plus a handful of smaller things, left inline below.

The gap: on a managed runtime installed with the devel extra, the new check can never fire, and comgr_matches_runtime comes back null rather than true on a healthy install. This PR fixes the discovery half of that problem, install_library_dirs now prefers the probe's recorded library_paths over re-deriving the layout, but the attribution half still compares those paths against a root that is the _rocm_sdk_devel directory, which they are not under. The unit tests do not catch it because their fixtures are built from a venv root that production never supplies.

The rest are documentation and description accuracy: two doc comments have drifted onto the wrong functions, docs/wsl.md now contradicts the registration table, and the description's CI section describes work that moved to #384.

One concern I chased and dropped: the claim that the venv-layout fallback had disappeared from examine's search. It has not, library_paths has been in the probe script since the initial import.

Comment thread crates/rocm-core/src/examine.rs Resolved
Comment thread crates/rocm-core/src/diagnose.rs Resolved
Comment thread crates/rocm-core/src/examine.rs Resolved
Comment thread crates/rocm-core/src/diagnose.rs Resolved
@volen-silo
volen-silo force-pushed the feat/report-comgr-copies branch from a6d7af6 to e33f8ee Compare October 2, 2026 14:07
@volen-silo
volen-silo force-pushed the feat/diagnose-comgr-conflict branch from 19e0fda to ec761a5 Compare October 2, 2026 14:56
@volen-silo
volen-silo force-pushed the feat/report-comgr-copies branch 2 times, most recently from ba1d7fb to 53b13d2 Compare October 5, 2026 06:38
@volen-silo
volen-silo force-pushed the feat/diagnose-comgr-conflict branch from ec761a5 to 3dfb896 Compare October 5, 2026 06:56
@volen-silo
volen-silo dismissed juhovainio’s stale review October 5, 2026 07:25

All five items addressed: attribution fixed from prefix-match to an exact directory-owner map (dir_owners), with the regression fixture rebuilt from the production sibling-directory layout and confirmed by mutation testing (reverted to the old prefix match, watched the test go red, reverted back, confirmed green and no residual diff); both doc-comment splices fixed to sit on their own functions; docs/wsl.md updated so it no longer contradicts the fix-18 registration; PR description corrected for examine-19 naming and CI attribution to #384; and the fix.rs/diagnose.rs remedy text now shares one constant, structurally removing the drift the e2e pin could miss. Full verification bar is clean: cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings, cargo clippy -p e2e-cucumber --test e2e -- -D warnings, cargo test --workspace --all-targets (45 binaries, 0 failures), and cargo xtask e2e -- -n skill (6 scenarios, 22 steps, 0 failures).

@volen-silo
volen-silo force-pushed the feat/diagnose-comgr-conflict branch from 3dfb896 to da83dcf Compare October 5, 2026 07:35

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed after the fixes for the last round — all 5 previously-flagged issues are genuinely fixed, including the owning_install_root attribution bug (this now does exact-directory attribution via dir_owners, not just discovery).

One new issue in the comgr/HIP generalization: probe_comgr documents the comgr and HIP searches as symmetric ("searched the same way, and for the same reason"), but the "N copies found; X would load" note is only emitted for comgr. The HIP branch only checks hip.is_empty() and never calls multi_copy_note for it, so a machine with multiple libamdhip64 copies gets no note, even though the equivalent comgr case does. Left as an inline comment below.

Comment thread crates/rocm-core/src/examine.rs Resolved
@volen-silo
volen-silo force-pushed the feat/report-comgr-copies branch from 53b13d2 to 13ed5d4 Compare October 5, 2026 09:42
@volen-silo
volen-silo force-pushed the feat/diagnose-comgr-conflict branch from da83dcf to 955608f Compare October 5, 2026 10:12
r0x0r
r0x0r previously requested changes Oct 5, 2026

@r0x0r r0x0r left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took another look at this at 955608fd. Last time I looked (at an older commit) I said nothing blocking survived; this round turned up four things worth fixing before merge, left inline below.

Two are catalog bookkeeping that nothing in CI will catch, and they're the cheapest to fix: reference.md's heading still says 24 failure modes when the table now has 25 rows, and SKILL.md's "the other 20 are print-only" is now 21. You did correctly bump fix.rs's assert_eq!(count, 25) and update the OS recap line, so these are just the two surfaces with no assertion behind them — rocm_doctor_skill.feature says outright that both are unparsed and can go stale.

The other two are substantive:

  • The new HIP e2e step panics on a state this PR deliberately introduces. assert_hip_selection_is_stated has only two branches, so "copies found, but none on a tier the loader consults" hits the else arm and panics — even though select_loader_copy returns None for exactly that case and your own unit test pins it for libamdhip64 specifically. The comgr twin right below it already has the third branch; the HIP copy just didn't get it.
  • fix-18's evidence line tells the user the finding describes their plain shell, but the selection it rests on is computed with the active managed runtime's directories searched first. find_library_copies' own doc comment says that order "is not this process's search order; it is a served child's." That matters because the remedy you hand the user is a shell export, which can't change an outcome measured in the served-child order.

A few things I chased and dropped, for what it's worth: owning_install_root attributes from the canonicalised real_path, so a /opt/rocm symlink resolves both libraries to the same owner; check_18_comgr_conflict and comgr_matches_runtime really do share one predicate now, so they can't contradict each other; and the generated export LD_LIBRARY_PATH=... in diagnose.rs does point at the runtime's own copy, matching the prose above it.

Separately — the five comments from the earlier, now-dismissed review never got replies, so I checked them against this head. Four are genuinely fixed (option (b) wording, the comgr_matches_runtime rule, the doubled ldconfig -p, and the non-adjacent root dedup, which now has a test using the exact ordering from the comment). The fifth is still open: install_library_dirs calls collect_sdk_library_paths unconditionally, so bare-metal /opt/rocm roots now also get bin and lib/rocm_sysdeps/lib probed where the old comgr_search_dirs only added lib/lib64. I don't think it's a defect — those hits land on the rocm-install tier, which select_loader_copy now excludes from "would load" — but it was asked as a question and still deserves an answer in the PR text.

Comment thread skills/rocm-doctor/reference.md Resolved
Comment thread crates/rocm-core/src/fix.rs Resolved
Comment thread crates/rocm-core/src/diagnose.rs Resolved
@volen-silo
volen-silo force-pushed the feat/report-comgr-copies branch from 13ed5d4 to e7d40a3 Compare October 5, 2026 13:23

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I independently verified r0x0r's four open findings against the current diff — all four are real and still unaddressed, so I'm requesting changes to confirm them rather than leave them resting only on r0x0r's earlier review.

Separately: the earlier multi_copy_note/HIP-branch asymmetry I flagged is fixed, and fixed the way Eugene described — probe_comgr's HIP arm now calls the same note/selection path as the comgr arm, and the incidental .first() tier-selection bug is gone from production code too. No new issues beyond what's below.

The one I'd treat as the real blocker is the diagnose.rs:1498 one: the printed evidence line claims to describe "the environment as it stands outside the CLI's managed runtimes," but the conflict this check fires on can happen with an active managed runtime (the runtime ships its own HIP but not its own comgr). In that case the evidence may have come from the active-runtime tier, not the plain-shell tiers the sentence claims — and the fix's export LD_LIBRARY_PATH=... advice doesn't touch what rocm serve actually loads, so it could be inert for exactly the case it's describing.

Comment thread skills/rocm-doctor/reference.md Resolved
Comment thread crates/rocm-core/src/diagnose.rs Resolved
…s own

Builds on the copy search: with every copy of libamd_comgr known, the
catalog can say when the one that loads belongs to a different
installation than the HIP runtime that loads, and device code
compilation therefore fails with an error naming neither.

The rule is symmetric, and that is what makes it safe. The design note
for this entry proposed a special case -- "when the active runtime is
the managed runtime, the matching copy is the wheel copy" -- without
which it "fires on every healthy CLI installation". That is a patch over
a rule stated asymmetrically. Asking one question of both libraries
instead, does the libamd_comgr that would load come from the same
installation as the libamdhip64 that would load, makes every case fall
out of the rule: a healthy managed install takes both from the managed
runtime, so nothing differs and nothing is reported. A control that has
to be written as an exception is a rule that has not been stated
correctly yet.

That turns on attributing a copy to its installation correctly. A
managed runtime spreads its libraries across separate _rocm_sdk_*
packages that sit beside each other under one site-packages, not nested
under the runtime's own root at all, so ownership is decided by looking a
found file's directory up against the same directories
install_library_dirs already recorded for each root -- never by walking
up from the file, which would call each package its own install and fire
on the most common install we ship. That is the property the mutation
check targets.

Two further guards on firing: an unattributed copy is a gap in what the
search knows rather than a finding about the machine, and the runtime
must actually ship a copy of its own, or there is nothing to point the
user at.

The entry is print-only and ranks neither remedy. Removing a stack and
reordering the search path can each break a working Python environment,
and which is right depends on which stack the user means to keep.

The finding says out loud that it describes the environment outside the
managed runtimes. `rocm serve` puts a managed runtime's libraries first
on purpose and gets a different, correct answer, and a report that did
not name the environment it examined would read as a claim about one it
never looked at.

The rule above is now stated exactly once, as `comgr_matches_runtime` in
examine.rs, and both this entry and `rocm examine --json`'s field of the
same name call it rather than each restating the conditions -- so the
two surfaces cannot disagree about one machine, and a regression test
pins that. Fix option (b)'s advice matched an earlier, asymmetric
statement of the rule; it now says to prefer the wheel's own copy of
both libraries, which is what the symmetric rule actually asks the user
to do. The machine-readable inspection now carries the same
`install_root` attribution for the HIP runtime side that it already
carried for the code object manager side, since the conflict question is
about both libraries and a reader could not previously check either the
attribution or the selection for one of them.

Known roots are deduplicated across the whole list, not only where
repeats happen to land next to each other. `dedup_by` drops consecutive
duplicates, and these roots arrive from three independent sources -- the
active install, a sorted /opt scan, and the managed runtimes -- so the
active install colliding with a sibling was caught or missed depending on
where the path sorted. A root recorded twice can make a machine holding
one stack read as a machine holding two, which is the false conflict this
entry exists not to report.

hip_paths and hip_selected are `serde(default)` for the same reason the
comgr fields are: an examination read back over the remote path may have
been produced by an older CLI that never wrote them.

Once a managed runtime is found, its root resolves to its
`_rocm_sdk_devel` package directory, and the search used to guess a venv
layout (`root/lib/<python>/site-packages`) to find its sibling packages;
that guess does not hold for a real install, so the search now reads
back the library directories the SDK probe already recorded
(`library_paths`) instead of re-deriving them. This generalises the
walker introduced in #384 (`find_library_copies`/`install_library_dirs`)
so it serves the HIP side too, which is why `managed_runtime_roots` now
returns `library_paths` plural rather than a single `site_packages`
guess.

The AppPaths::discover data-dir fix and the @requires-os:linux platform
tags this commit originally carried for examine-17/18 belong with the
scenario they correct, which #384 introduced -- moved there so the
parent doesn't depend on the child to pass its own CI, and this commit
now only rebases on top of that fix rather than restating it.

Sweeping the same gap once it was visible: examine-17 (the
host-independent code-object-manager scenario, renumbered by #384's
rebase) asserts `hip_paths`/`hip_selected` the same way it asserts
`comgr_paths`/`comgr_selected`, and the empty branch had the identical
problem #384's review flagged for the comgr side -- `hip_paths: []` /
`hip_selected: null` also hold by nothing more than `Examination`'s own
defaults, so the assertion could not tell "probed, found none" from
"never probed". `probe_comgr` now pushes a "no libamdhip64 found" note
the same way it already did for libamd_comgr, and the HIP assertion now
requires it in the empty branch. Verified by temporarily suppressing the
note and watching the scenario go red, then restoring it.

Review: the attribution half still did not survive a realistic managed
runtime. `known_install_roots` registers a managed runtime by
`candidate.root_path` alone -- `_rocm_sdk_devel` when the `devel` extra
is installed -- and `libamd_comgr`/`libamdhip64` live in the sibling
`_rocm_sdk_core` package, which sits *beside* `root_path`, not under it.
A string-prefix match against the bare root therefore attributed nothing
on exactly this shape: both libraries came back `install_root: ""`,
`comgr_matches_runtime` hit its empty-root guard and returned `None`
instead of `true`, and `check_18_comgr_conflict` could not fire for that
runtime at all. The two unit tests meant to catch this
(`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime`,
`a_healthy_managed_installation_raises_no_report`) built their fixtures
from a venv root (`/data/runtimes/therock/default`,
`.../lib/python3.12/site-packages/_rocm_sdk_core/...`) that production
never produces, so they passed regardless.

Fixed at the root: `install_library_dirs` already walks, for every known
root, exactly the directories that root's copies live in (that is the
discovery-side fix this commit already made). `find_library_copies` now
builds a directory -> owning-root map from that same walk and attributes
each found file by looking its directory up in it, rather than by
string-prefix match against the bare root. A sibling package's directory
is simply one more entry in the map, keyed to the runtime that recorded
it, so it attributes correctly without being nested under anything.
`owning_install_root` and `record_library_copy` take that map instead of
the bare root list; `the_nearest_enclosing_installation_claims_a_library`
is removed because the "longest prefix wins" ambiguity it guarded against
cannot arise under exact directory lookup. Both fixture tests above were
rebuilt from the real installer shape (sibling `_rocm_sdk_*` packages
under one `site-packages`, matching `apps/rocm/src/therock.rs`'s
`ROCM_SDK_PROBE_SCRIPT`), and
`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` was
confirmed to fail against the old string-prefix attribution before
passing against the fix.

`find_library_copies` also gained the `active_runtime_dirs` parameter
runtime's own library directories are now searched first, ahead of the
ambient `LD_LIBRARY_PATH`, for both comgr and HIP, since `rocm serve`
prepends them for both libraries equally. The loader cache and the
directory walk are now gathered once in `probe_comgr` and passed to both
the comgr and HIP searches, rather than each search reading
`LD_LIBRARY_PATH`, spawning `ldconfig -p`, and walking every managed
runtime's directories again on every invocation -- neither depends on
which library is being searched for, so the duplication cost an extra
subprocess and directory walk on every `rocm examine`/`rocm diagnose`
for no reason. `multi_copy_note` is split out as its own pure function
so the "N copies ... would load" note stays independently testable now
that `find_library_copies` only returns copies.

Two doc-comment splices, both from a missing blank `///` line between two
unrelated items: `check_18_comgr_conflict`'s doc paragraph had absorbed
check_17's above it, leaving check_17 undocumented; the same shape left
`known_install_roots` undocumented under a paragraph describing
`dedup_roots_keeping_first`. Both are split back onto the function they
actually describe.

`docs/wsl.md` said `fix-19-shm-too-small` was the only non-WSL entry that
applies on WSL; registering `check_18_comgr_conflict` for `["linux",
"wsl"]` made that false the moment this entry existed. Updated to name
both.

The "Neither option is recommended..." sentence was stated independently
in `fix.rs`'s static recipe and `diagnose.rs`'s dynamically-generated one,
in slightly different words -- already drifted, and only the `fix.rs`
copy was pinned by an e2e assertion, so the `diagnose.rs` copy could drift
further with nothing to catch it. Both now read
`fix::COMGR_CONFLICT_NEITHER_OPTION_RECOMMENDED`, so they cannot
disagree.

Not changed: `fix.rs`'s option (b) wording and `comgr_matches_runtime`'s
parity with `check_18_comgr_conflict` were flagged in an earlier review
round and were already correct by the time this round started --
verified against the current code rather than re-fixed.

Rebase onto #384 and a second symmetry gap: #384's own review fixed
`comgr_selected` picking `copies.first()` regardless of tier -- a
`rocm-install`/`managed-runtime`-only hit used to be reported as "the
one that would load" purely because it sorted first, when neither
source is one the loader consults on its own. Rebasing that fix under
this entry's HIP-runtime work surfaced a second instance of the same
asymmetry a human reviewer caught directly on this branch:
`probe_comgr`'s doc comment claimed comgr and HIP were searched and
reported identically, but only comgr ever got a `multi_copy_note` call;
the HIP branch only checked `hip.is_empty()` and pushed nothing for the
"found, but only in a directory the loader does not consult" case the
comgr side had just been fixed to report.

Fixed by stating the tier rule once and applying it to both prefixes
rather than fixing it twice: `select_loader_copy` replaces
`copies.first()` with a search restricted to `LOADER_PATH_SOURCES`
(`active-runtime`, `ld-library-path`, `loader-cache` -- the tiers the
loader itself consults), and `copies_note` replaces `multi_copy_note`
with the three-way note (no copies / more than one with a winner /
one-or-more with none selected) that a `rocm-install`-only hit now
needs. `probe_comgr` calls both exactly once for comgr and once for HIP,
so there is one call site per library rather than one function's logic
duplicated by hand into the other's branch, and the doc comment now says
what the code does instead of what it used to do before #384's fix
landed.

Pinned by `a_search_dir_only_hit_is_not_selected_for_either_prefix`,
which runs the same assertions for `COMGR_LIB_PREFIX` and
`HIP_RUNTIME_LIB_PREFIX` in one test rather than two, so the rule is
proven to apply to both rather than merely written to apply to both.
The existing `loader_cache_outranks_search_dirs` and
`the_active_runtimes_own_copy_outranks_the_ambient_environment`/
`ld_library_path_outranks_loader_cache_and_search_dirs` tests were
carried through the rebase onto `find_library_copies`/
`select_loader_copy` rather than dropped, since they pin mutation-tested
tier-ordering bugs #384 already fixed once.

`comgr_search_dirs_labels_each_tier`, which tested the now-deleted
`comgr_search_dirs_in`, is removed as redundant: `install_library_dirs`'s
own tier-labelling is already covered by
`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` and
`a_library_no_installation_claims_is_attributed_to_none`.

Verified: `cargo fmt --all -- --check`, `cargo clippy --workspace
--all-targets -- -D warnings`, `cargo clippy -p e2e-cucumber --test e2e
-- -D warnings`, and `cargo test --workspace --all-targets` all pass
(one `xtask::workflow_contract` test flaked under full-suite parallel
load on a timing assertion against a real `rocm-smi` subprocess call and
passed cleanly in isolation and on a full-suite rerun; unrelated to this
change). Cross-compiled the whole workspace, tests included, for
`x86_64-pc-windows-gnu` under `-D warnings` to confirm no code is left
unix-only-reachable and therefore dead on Windows. `cargo xtask e2e --
-n skill` passes all 6 scenarios, since this round also touches
`skills/rocm-doctor/reference.md`.

Second review round, two independent reviewers filing the same four findings
(verified rather than merely accepted):

fix-18's evidence line claimed to describe "the environment as it stands
outside the CLI's managed runtimes" unconditionally, even when a selection
actually came from the active-runtime tier -- a served child's search order,
not a plain shell's. That also means the `export LD_LIBRARY_PATH=...` remedy
can be inert for exactly that case, since `lemonade_process_environment_vars`
prepends the runtime's own directories ahead of whatever the remedy exports.
Rewording the evidence line is the easy half; the detection logic that
decides which sentence to print cannot move to plain-shell-only without
reintroducing the false positives/negatives `comgr_matches_runtime` and
`a_healthy_managed_installation_raises_no_report` were written to rule out,
since that guarantee depends on both selections already being
active-runtime-tier-aware. So `comgr_matches_runtime` is untouched;
`check_18_comgr_conflict` now checks whether either selected copy's source is
`active-runtime` and, when it is, swaps in an honest sentence plus a new
remedy note stating the export may not reach a served child and pointing at
reinstalling/repairing that runtime instead. Pinned by
`an_active_runtime_s_own_hip_without_its_own_comgr_notes_the_scope_limit`,
mutation-tested by forcing the branch condition false and watching the new
test fail on the stale sentence, then reverting.

`assert_hip_selection_is_stated`, the e2e HIP twin of the comgr selection
assertion, had only two branches, so "copies found, but none on a tier the
loader consults" -- `select_loader_copy` returning `None` on purpose, already
pinned for `libamdhip64` by
`a_search_dir_only_hit_is_not_selected_for_either_prefix` -- fell into the
`else` arm and asserted against a selected copy that does not exist. Added
the missing third branch, mirroring `assert_comgr_selection_is_stated`
exactly, including its `LOADER_PATH_SOURCES` check on the selected copy's
source, which the HIP step's final arm was also missing. Hoisted the shared
`LOADER_PATH_SOURCES` const the two steps now both use so they cannot drift
apart again.

`skills/rocm-doctor/reference.md`'s "Closed catalog (24 failure modes)"
heading and `SKILL.md`'s "the other 20 are print-only" line had both gone
stale against the table's 25 rows and 4 auto-applicable fixes
(`exactly_the_four_known_fixes_are_auto`); fixed to 25 and 21.
`rocm_doctor_skill.feature` deliberately leaves both counts unparsed against
the live CLI, and that scope decision stands -- the CLI is not wired to
report "how many rows exist" as a fact an agent reads. What it does not rule
out is checking the two numbers against each other, so
`tests/e2e-cucumber/tests/skill_reference.rs` (plain `cargo test`, no binary
needed) gained `free_standing_catalog_counts_match_the_table`, which parses
both numbers back out of the two docs and asserts them against the table's
own row and auto-fix counts. Mutation-tested by reverting each number one at
a time and confirming the new test goes red, then restoring it.

Open question from the first round, never answered: does
`install_library_dirs` calling `collect_sdk_library_paths` unconditionally
widen what a bare-metal `/opt/rocm` root probes in a way that matters?
Traced it rather than taking the earlier answer on faith: every directory
`install_library_dirs` adds for a root carries that root's own source
(`rocm-install` or `managed-runtime`, never a loader-consulted tier), and
`find_library_copies` only records a copy under that source when no
loader-consulted tier (`active-runtime`/`ld-library-path`/`loader-cache`)
already found the same path first -- so a wider probe can only add more
`rocm-install`-sourced evidence, never promote a path onto a tier
`select_loader_copy` would pick. The one place a wider probe does change an
answer is `comgr_matches_runtime`'s "does the runtime ship its own copy"
check, and there a new hit can only turn a false conflict report into `None`
(not enough evidence), which is strictly more correct, not a new risk. The
reviewer's reasoning holds; recorded here since the first round left it
unanswered.

Re-verified after this round's changes: `cargo fmt --all -- --check`, `cargo
clippy --workspace --all-targets -- -D warnings`, `cargo clippy -p
e2e-cucumber --test e2e -- -D warnings`, `cargo test --workspace
--all-targets` (251 passed, 0 failed across 33 binaries), and
`RUSTFLAGS="-D warnings" cargo check --workspace --all-targets --target
x86_64-pc-windows-gnu` all pass clean. `cargo xtask e2e -- -n skill` passes
all 6 scenarios again; `cargo xtask e2e -- -n examine-18` passes its 5 steps,
confirming the new HIP e2e branch does not break the existing scenario.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
@volen-silo
volen-silo force-pushed the feat/diagnose-comgr-conflict branch from 955608f to 0ed84ba Compare October 5, 2026 15:00
@volen-silo
volen-silo dismissed stale reviews from r0x0r and juhovainio October 5, 2026 15:07

All four findings addressed in the amended commit: fix-18's evidence/remedy made conditional on active-runtime involvement (not just reworded), the HIP e2e step's missing third branch added (unified with the comgr step via select_loader_copy/copies_note), reference.md/SKILL.md counts fixed to 25/21 with a new drift-guard test, and the install_library_dirs scan-widening question traced and confirmed safe. Replied inline to each; see the PR description's "Review remediation, round two" section.

@volen-silo

Copy link
Copy Markdown
Collaborator Author

Addressed the second review round (r0x0r + juhovainio, same four findings) in 955608fd:

fix-18 evidence/remedy (#4184271400, #4185096435) — made the evidence line conditional on whether either selected copy came from the active-runtime tier, rather than rewording it unconditionally. When it did, a new sentence states the served-child caveat and a new fix note says the export remedy may not reach it, pointing at reinstalling/repairing the runtime instead.

HIP e2e missing third branch (#4184271391, #4185096422) — unified the comgr/HIP selection+note logic (select_loader_copy + copies_note) instead of copy-pasting the comgr branch, since the old pick was tier-blind. Both e2e steps now share LOADER_PATH_SOURCES and the same three branches.

Stale catalog counts (#4184271375, #4184271383, #4185096413) — fixed to 25/21, and added free_standing_catalog_counts_match_the_table to tests/e2e-cucumber/tests/skill_reference.rs as a cheap drift guard.

Open question on install_library_dirs's widened scan (#4121402367, carried over from the first review round) — traced the full chain and confirmed it's safe: a wider probe can only add more rocm-install-sourced evidence, which select_loader_copy already excludes from "would load", and the one place it affects an answer (comgr_matches_runtime) can only become more conservative, never less. Answered in the PR description under "Open question: does widening the bare-metal scan matter?".

Full detail is in the commit message's "Review remediation, round two" section and the PR description. Re-ran the full verification bar plus cargo xtask e2e -- -n skill and cargo xtask e2e -- -n examine-18, all clean. Both CHANGES_REQUESTED reviews dismissed.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked all three fixes against the actual current code, not just the replies:

  • diagnose.rs's comgr-conflict evidence line is now conditional on whether either selected copy came from the active-runtime tier, with the honest sentence either way plus an added fix note that the export may not reach a served child. Pinned by a new unit test.
  • examine_steps.rs's HIP-selection assertion now has the missing third branch for "copies found, none selected" (select_loader_copy returning None on purpose), matching the comgr sibling.
  • The catalog heading and SKILL.md's print-only count both now match the live table (25 rows, 4 auto-applicable = 21), backed by a new cargo test-level drift guard so this can't go stale again silently.

Looks good to merge.

@volen-silo
volen-silo merged commit 66021dc into feat/report-comgr-copies Oct 5, 2026
20 checks passed
@volen-silo
volen-silo deleted the feat/diagnose-comgr-conflict branch October 5, 2026 15:59
volen-silo added a commit that referenced this pull request Oct 5, 2026
…s own (#385)

Builds on the copy search: with every copy of libamd_comgr known, the
catalog can say when the one that loads belongs to a different
installation than the HIP runtime that loads, and device code
compilation therefore fails with an error naming neither.

The rule is symmetric, and that is what makes it safe. The design note
for this entry proposed a special case -- "when the active runtime is
the managed runtime, the matching copy is the wheel copy" -- without
which it "fires on every healthy CLI installation". That is a patch over
a rule stated asymmetrically. Asking one question of both libraries
instead, does the libamd_comgr that would load come from the same
installation as the libamdhip64 that would load, makes every case fall
out of the rule: a healthy managed install takes both from the managed
runtime, so nothing differs and nothing is reported. A control that has
to be written as an exception is a rule that has not been stated
correctly yet.

That turns on attributing a copy to its installation correctly. A
managed runtime spreads its libraries across separate _rocm_sdk_*
packages that sit beside each other under one site-packages, not nested
under the runtime's own root at all, so ownership is decided by looking a
found file's directory up against the same directories
install_library_dirs already recorded for each root -- never by walking
up from the file, which would call each package its own install and fire
on the most common install we ship. That is the property the mutation
check targets.

Two further guards on firing: an unattributed copy is a gap in what the
search knows rather than a finding about the machine, and the runtime
must actually ship a copy of its own, or there is nothing to point the
user at.

The entry is print-only and ranks neither remedy. Removing a stack and
reordering the search path can each break a working Python environment,
and which is right depends on which stack the user means to keep.

The finding says out loud that it describes the environment outside the
managed runtimes. `rocm serve` puts a managed runtime's libraries first
on purpose and gets a different, correct answer, and a report that did
not name the environment it examined would read as a claim about one it
never looked at.

The rule above is now stated exactly once, as `comgr_matches_runtime` in
examine.rs, and both this entry and `rocm examine --json`'s field of the
same name call it rather than each restating the conditions -- so the
two surfaces cannot disagree about one machine, and a regression test
pins that. Fix option (b)'s advice matched an earlier, asymmetric
statement of the rule; it now says to prefer the wheel's own copy of
both libraries, which is what the symmetric rule actually asks the user
to do. The machine-readable inspection now carries the same
`install_root` attribution for the HIP runtime side that it already
carried for the code object manager side, since the conflict question is
about both libraries and a reader could not previously check either the
attribution or the selection for one of them.

Known roots are deduplicated across the whole list, not only where
repeats happen to land next to each other. `dedup_by` drops consecutive
duplicates, and these roots arrive from three independent sources -- the
active install, a sorted /opt scan, and the managed runtimes -- so the
active install colliding with a sibling was caught or missed depending on
where the path sorted. A root recorded twice can make a machine holding
one stack read as a machine holding two, which is the false conflict this
entry exists not to report.

hip_paths and hip_selected are `serde(default)` for the same reason the
comgr fields are: an examination read back over the remote path may have
been produced by an older CLI that never wrote them.

Once a managed runtime is found, its root resolves to its
`_rocm_sdk_devel` package directory, and the search used to guess a venv
layout (`root/lib/<python>/site-packages`) to find its sibling packages;
that guess does not hold for a real install, so the search now reads
back the library directories the SDK probe already recorded
(`library_paths`) instead of re-deriving them. This generalises the
walker introduced in #384 (`find_library_copies`/`install_library_dirs`)
so it serves the HIP side too, which is why `managed_runtime_roots` now
returns `library_paths` plural rather than a single `site_packages`
guess.

The AppPaths::discover data-dir fix and the @requires-os:linux platform
tags this commit originally carried for examine-17/18 belong with the
scenario they correct, which #384 introduced -- moved there so the
parent doesn't depend on the child to pass its own CI, and this commit
now only rebases on top of that fix rather than restating it.

Sweeping the same gap once it was visible: examine-17 (the
host-independent code-object-manager scenario, renumbered by #384's
rebase) asserts `hip_paths`/`hip_selected` the same way it asserts
`comgr_paths`/`comgr_selected`, and the empty branch had the identical
problem #384's review flagged for the comgr side -- `hip_paths: []` /
`hip_selected: null` also hold by nothing more than `Examination`'s own
defaults, so the assertion could not tell "probed, found none" from
"never probed". `probe_comgr` now pushes a "no libamdhip64 found" note
the same way it already did for libamd_comgr, and the HIP assertion now
requires it in the empty branch. Verified by temporarily suppressing the
note and watching the scenario go red, then restoring it.

Review: the attribution half still did not survive a realistic managed
runtime. `known_install_roots` registers a managed runtime by
`candidate.root_path` alone -- `_rocm_sdk_devel` when the `devel` extra
is installed -- and `libamd_comgr`/`libamdhip64` live in the sibling
`_rocm_sdk_core` package, which sits *beside* `root_path`, not under it.
A string-prefix match against the bare root therefore attributed nothing
on exactly this shape: both libraries came back `install_root: ""`,
`comgr_matches_runtime` hit its empty-root guard and returned `None`
instead of `true`, and `check_18_comgr_conflict` could not fire for that
runtime at all. The two unit tests meant to catch this
(`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime`,
`a_healthy_managed_installation_raises_no_report`) built their fixtures
from a venv root (`/data/runtimes/therock/default`,
`.../lib/python3.12/site-packages/_rocm_sdk_core/...`) that production
never produces, so they passed regardless.

Fixed at the root: `install_library_dirs` already walks, for every known
root, exactly the directories that root's copies live in (that is the
discovery-side fix this commit already made). `find_library_copies` now
builds a directory -> owning-root map from that same walk and attributes
each found file by looking its directory up in it, rather than by
string-prefix match against the bare root. A sibling package's directory
is simply one more entry in the map, keyed to the runtime that recorded
it, so it attributes correctly without being nested under anything.
`owning_install_root` and `record_library_copy` take that map instead of
the bare root list; `the_nearest_enclosing_installation_claims_a_library`
is removed because the "longest prefix wins" ambiguity it guarded against
cannot arise under exact directory lookup. Both fixture tests above were
rebuilt from the real installer shape (sibling `_rocm_sdk_*` packages
under one `site-packages`, matching `apps/rocm/src/therock.rs`'s
`ROCM_SDK_PROBE_SCRIPT`), and
`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` was
confirmed to fail against the old string-prefix attribution before
passing against the fix.

`find_library_copies` also gained the `active_runtime_dirs` parameter
runtime's own library directories are now searched first, ahead of the
ambient `LD_LIBRARY_PATH`, for both comgr and HIP, since `rocm serve`
prepends them for both libraries equally. The loader cache and the
directory walk are now gathered once in `probe_comgr` and passed to both
the comgr and HIP searches, rather than each search reading
`LD_LIBRARY_PATH`, spawning `ldconfig -p`, and walking every managed
runtime's directories again on every invocation -- neither depends on
which library is being searched for, so the duplication cost an extra
subprocess and directory walk on every `rocm examine`/`rocm diagnose`
for no reason. `multi_copy_note` is split out as its own pure function
so the "N copies ... would load" note stays independently testable now
that `find_library_copies` only returns copies.

Two doc-comment splices, both from a missing blank `///` line between two
unrelated items: `check_18_comgr_conflict`'s doc paragraph had absorbed
check_17's above it, leaving check_17 undocumented; the same shape left
`known_install_roots` undocumented under a paragraph describing
`dedup_roots_keeping_first`. Both are split back onto the function they
actually describe.

`docs/wsl.md` said `fix-19-shm-too-small` was the only non-WSL entry that
applies on WSL; registering `check_18_comgr_conflict` for `["linux",
"wsl"]` made that false the moment this entry existed. Updated to name
both.

The "Neither option is recommended..." sentence was stated independently
in `fix.rs`'s static recipe and `diagnose.rs`'s dynamically-generated one,
in slightly different words -- already drifted, and only the `fix.rs`
copy was pinned by an e2e assertion, so the `diagnose.rs` copy could drift
further with nothing to catch it. Both now read
`fix::COMGR_CONFLICT_NEITHER_OPTION_RECOMMENDED`, so they cannot
disagree.

Not changed: `fix.rs`'s option (b) wording and `comgr_matches_runtime`'s
parity with `check_18_comgr_conflict` were flagged in an earlier review
round and were already correct by the time this round started --
verified against the current code rather than re-fixed.

Rebase onto #384 and a second symmetry gap: #384's own review fixed
`comgr_selected` picking `copies.first()` regardless of tier -- a
`rocm-install`/`managed-runtime`-only hit used to be reported as "the
one that would load" purely because it sorted first, when neither
source is one the loader consults on its own. Rebasing that fix under
this entry's HIP-runtime work surfaced a second instance of the same
asymmetry a human reviewer caught directly on this branch:
`probe_comgr`'s doc comment claimed comgr and HIP were searched and
reported identically, but only comgr ever got a `multi_copy_note` call;
the HIP branch only checked `hip.is_empty()` and pushed nothing for the
"found, but only in a directory the loader does not consult" case the
comgr side had just been fixed to report.

Fixed by stating the tier rule once and applying it to both prefixes
rather than fixing it twice: `select_loader_copy` replaces
`copies.first()` with a search restricted to `LOADER_PATH_SOURCES`
(`active-runtime`, `ld-library-path`, `loader-cache` -- the tiers the
loader itself consults), and `copies_note` replaces `multi_copy_note`
with the three-way note (no copies / more than one with a winner /
one-or-more with none selected) that a `rocm-install`-only hit now
needs. `probe_comgr` calls both exactly once for comgr and once for HIP,
so there is one call site per library rather than one function's logic
duplicated by hand into the other's branch, and the doc comment now says
what the code does instead of what it used to do before #384's fix
landed.

Pinned by `a_search_dir_only_hit_is_not_selected_for_either_prefix`,
which runs the same assertions for `COMGR_LIB_PREFIX` and
`HIP_RUNTIME_LIB_PREFIX` in one test rather than two, so the rule is
proven to apply to both rather than merely written to apply to both.
The existing `loader_cache_outranks_search_dirs` and
`the_active_runtimes_own_copy_outranks_the_ambient_environment`/
`ld_library_path_outranks_loader_cache_and_search_dirs` tests were
carried through the rebase onto `find_library_copies`/
`select_loader_copy` rather than dropped, since they pin mutation-tested
tier-ordering bugs #384 already fixed once.

`comgr_search_dirs_labels_each_tier`, which tested the now-deleted
`comgr_search_dirs_in`, is removed as redundant: `install_library_dirs`'s
own tier-labelling is already covered by
`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` and
`a_library_no_installation_claims_is_attributed_to_none`.

Verified: `cargo fmt --all -- --check`, `cargo clippy --workspace
--all-targets -- -D warnings`, `cargo clippy -p e2e-cucumber --test e2e
-- -D warnings`, and `cargo test --workspace --all-targets` all pass
(one `xtask::workflow_contract` test flaked under full-suite parallel
load on a timing assertion against a real `rocm-smi` subprocess call and
passed cleanly in isolation and on a full-suite rerun; unrelated to this
change). Cross-compiled the whole workspace, tests included, for
`x86_64-pc-windows-gnu` under `-D warnings` to confirm no code is left
unix-only-reachable and therefore dead on Windows. `cargo xtask e2e --
-n skill` passes all 6 scenarios, since this round also touches
`skills/rocm-doctor/reference.md`.

Second review round, two independent reviewers filing the same four findings
(verified rather than merely accepted):

fix-18's evidence line claimed to describe "the environment as it stands
outside the CLI's managed runtimes" unconditionally, even when a selection
actually came from the active-runtime tier -- a served child's search order,
not a plain shell's. That also means the `export LD_LIBRARY_PATH=...` remedy
can be inert for exactly that case, since `lemonade_process_environment_vars`
prepends the runtime's own directories ahead of whatever the remedy exports.
Rewording the evidence line is the easy half; the detection logic that
decides which sentence to print cannot move to plain-shell-only without
reintroducing the false positives/negatives `comgr_matches_runtime` and
`a_healthy_managed_installation_raises_no_report` were written to rule out,
since that guarantee depends on both selections already being
active-runtime-tier-aware. So `comgr_matches_runtime` is untouched;
`check_18_comgr_conflict` now checks whether either selected copy's source is
`active-runtime` and, when it is, swaps in an honest sentence plus a new
remedy note stating the export may not reach a served child and pointing at
reinstalling/repairing that runtime instead. Pinned by
`an_active_runtime_s_own_hip_without_its_own_comgr_notes_the_scope_limit`,
mutation-tested by forcing the branch condition false and watching the new
test fail on the stale sentence, then reverting.

`assert_hip_selection_is_stated`, the e2e HIP twin of the comgr selection
assertion, had only two branches, so "copies found, but none on a tier the
loader consults" -- `select_loader_copy` returning `None` on purpose, already
pinned for `libamdhip64` by
`a_search_dir_only_hit_is_not_selected_for_either_prefix` -- fell into the
`else` arm and asserted against a selected copy that does not exist. Added
the missing third branch, mirroring `assert_comgr_selection_is_stated`
exactly, including its `LOADER_PATH_SOURCES` check on the selected copy's
source, which the HIP step's final arm was also missing. Hoisted the shared
`LOADER_PATH_SOURCES` const the two steps now both use so they cannot drift
apart again.

`skills/rocm-doctor/reference.md`'s "Closed catalog (24 failure modes)"
heading and `SKILL.md`'s "the other 20 are print-only" line had both gone
stale against the table's 25 rows and 4 auto-applicable fixes
(`exactly_the_four_known_fixes_are_auto`); fixed to 25 and 21.
`rocm_doctor_skill.feature` deliberately leaves both counts unparsed against
the live CLI, and that scope decision stands -- the CLI is not wired to
report "how many rows exist" as a fact an agent reads. What it does not rule
out is checking the two numbers against each other, so
`tests/e2e-cucumber/tests/skill_reference.rs` (plain `cargo test`, no binary
needed) gained `free_standing_catalog_counts_match_the_table`, which parses
both numbers back out of the two docs and asserts them against the table's
own row and auto-fix counts. Mutation-tested by reverting each number one at
a time and confirming the new test goes red, then restoring it.

Open question from the first round, never answered: does
`install_library_dirs` calling `collect_sdk_library_paths` unconditionally
widen what a bare-metal `/opt/rocm` root probes in a way that matters?
Traced it rather than taking the earlier answer on faith: every directory
`install_library_dirs` adds for a root carries that root's own source
(`rocm-install` or `managed-runtime`, never a loader-consulted tier), and
`find_library_copies` only records a copy under that source when no
loader-consulted tier (`active-runtime`/`ld-library-path`/`loader-cache`)
already found the same path first -- so a wider probe can only add more
`rocm-install`-sourced evidence, never promote a path onto a tier
`select_loader_copy` would pick. The one place a wider probe does change an
answer is `comgr_matches_runtime`'s "does the runtime ship its own copy"
check, and there a new hit can only turn a false conflict report into `None`
(not enough evidence), which is strictly more correct, not a new risk. The
reviewer's reasoning holds; recorded here since the first round left it
unanswered.

Re-verified after this round's changes: `cargo fmt --all -- --check`, `cargo
clippy --workspace --all-targets -- -D warnings`, `cargo clippy -p
e2e-cucumber --test e2e -- -D warnings`, `cargo test --workspace
--all-targets` (251 passed, 0 failed across 33 binaries), and
`RUSTFLAGS="-D warnings" cargo check --workspace --all-targets --target
x86_64-pc-windows-gnu` all pass clean. `cargo xtask e2e -- -n skill` passes
all 6 scenarios again; `cargo xtask e2e -- -n examine-18` passes its 5 steps,
confirming the new HIP e2e branch does not break the existing scenario.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
volen-silo added a commit that referenced this pull request Oct 5, 2026
…s own (#385)

Builds on the copy search: with every copy of libamd_comgr known, the
catalog can say when the one that loads belongs to a different
installation than the HIP runtime that loads, and device code
compilation therefore fails with an error naming neither.

The rule is symmetric, and that is what makes it safe. The design note
for this entry proposed a special case -- "when the active runtime is
the managed runtime, the matching copy is the wheel copy" -- without
which it "fires on every healthy CLI installation". That is a patch over
a rule stated asymmetrically. Asking one question of both libraries
instead, does the libamd_comgr that would load come from the same
installation as the libamdhip64 that would load, makes every case fall
out of the rule: a healthy managed install takes both from the managed
runtime, so nothing differs and nothing is reported. A control that has
to be written as an exception is a rule that has not been stated
correctly yet.

That turns on attributing a copy to its installation correctly. A
managed runtime spreads its libraries across separate _rocm_sdk_*
packages that sit beside each other under one site-packages, not nested
under the runtime's own root at all, so ownership is decided by looking a
found file's directory up against the same directories
install_library_dirs already recorded for each root -- never by walking
up from the file, which would call each package its own install and fire
on the most common install we ship. That is the property the mutation
check targets.

Two further guards on firing: an unattributed copy is a gap in what the
search knows rather than a finding about the machine, and the runtime
must actually ship a copy of its own, or there is nothing to point the
user at.

The entry is print-only and ranks neither remedy. Removing a stack and
reordering the search path can each break a working Python environment,
and which is right depends on which stack the user means to keep.

The finding says out loud that it describes the environment outside the
managed runtimes. `rocm serve` puts a managed runtime's libraries first
on purpose and gets a different, correct answer, and a report that did
not name the environment it examined would read as a claim about one it
never looked at.

The rule above is now stated exactly once, as `comgr_matches_runtime` in
examine.rs, and both this entry and `rocm examine --json`'s field of the
same name call it rather than each restating the conditions -- so the
two surfaces cannot disagree about one machine, and a regression test
pins that. Fix option (b)'s advice matched an earlier, asymmetric
statement of the rule; it now says to prefer the wheel's own copy of
both libraries, which is what the symmetric rule actually asks the user
to do. The machine-readable inspection now carries the same
`install_root` attribution for the HIP runtime side that it already
carried for the code object manager side, since the conflict question is
about both libraries and a reader could not previously check either the
attribution or the selection for one of them.

Known roots are deduplicated across the whole list, not only where
repeats happen to land next to each other. `dedup_by` drops consecutive
duplicates, and these roots arrive from three independent sources -- the
active install, a sorted /opt scan, and the managed runtimes -- so the
active install colliding with a sibling was caught or missed depending on
where the path sorted. A root recorded twice can make a machine holding
one stack read as a machine holding two, which is the false conflict this
entry exists not to report.

hip_paths and hip_selected are `serde(default)` for the same reason the
comgr fields are: an examination read back over the remote path may have
been produced by an older CLI that never wrote them.

Once a managed runtime is found, its root resolves to its
`_rocm_sdk_devel` package directory, and the search used to guess a venv
layout (`root/lib/<python>/site-packages`) to find its sibling packages;
that guess does not hold for a real install, so the search now reads
back the library directories the SDK probe already recorded
(`library_paths`) instead of re-deriving them. This generalises the
walker introduced in #384 (`find_library_copies`/`install_library_dirs`)
so it serves the HIP side too, which is why `managed_runtime_roots` now
returns `library_paths` plural rather than a single `site_packages`
guess.

The AppPaths::discover data-dir fix and the @requires-os:linux platform
tags this commit originally carried for examine-17/18 belong with the
scenario they correct, which #384 introduced -- moved there so the
parent doesn't depend on the child to pass its own CI, and this commit
now only rebases on top of that fix rather than restating it.

Sweeping the same gap once it was visible: examine-17 (the
host-independent code-object-manager scenario, renumbered by #384's
rebase) asserts `hip_paths`/`hip_selected` the same way it asserts
`comgr_paths`/`comgr_selected`, and the empty branch had the identical
problem #384's review flagged for the comgr side -- `hip_paths: []` /
`hip_selected: null` also hold by nothing more than `Examination`'s own
defaults, so the assertion could not tell "probed, found none" from
"never probed". `probe_comgr` now pushes a "no libamdhip64 found" note
the same way it already did for libamd_comgr, and the HIP assertion now
requires it in the empty branch. Verified by temporarily suppressing the
note and watching the scenario go red, then restoring it.

Review: the attribution half still did not survive a realistic managed
runtime. `known_install_roots` registers a managed runtime by
`candidate.root_path` alone -- `_rocm_sdk_devel` when the `devel` extra
is installed -- and `libamd_comgr`/`libamdhip64` live in the sibling
`_rocm_sdk_core` package, which sits *beside* `root_path`, not under it.
A string-prefix match against the bare root therefore attributed nothing
on exactly this shape: both libraries came back `install_root: ""`,
`comgr_matches_runtime` hit its empty-root guard and returned `None`
instead of `true`, and `check_18_comgr_conflict` could not fire for that
runtime at all. The two unit tests meant to catch this
(`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime`,
`a_healthy_managed_installation_raises_no_report`) built their fixtures
from a venv root (`/data/runtimes/therock/default`,
`.../lib/python3.12/site-packages/_rocm_sdk_core/...`) that production
never produces, so they passed regardless.

Fixed at the root: `install_library_dirs` already walks, for every known
root, exactly the directories that root's copies live in (that is the
discovery-side fix this commit already made). `find_library_copies` now
builds a directory -> owning-root map from that same walk and attributes
each found file by looking its directory up in it, rather than by
string-prefix match against the bare root. A sibling package's directory
is simply one more entry in the map, keyed to the runtime that recorded
it, so it attributes correctly without being nested under anything.
`owning_install_root` and `record_library_copy` take that map instead of
the bare root list; `the_nearest_enclosing_installation_claims_a_library`
is removed because the "longest prefix wins" ambiguity it guarded against
cannot arise under exact directory lookup. Both fixture tests above were
rebuilt from the real installer shape (sibling `_rocm_sdk_*` packages
under one `site-packages`, matching `apps/rocm/src/therock.rs`'s
`ROCM_SDK_PROBE_SCRIPT`), and
`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` was
confirmed to fail against the old string-prefix attribution before
passing against the fix.

`find_library_copies` also gained the `active_runtime_dirs` parameter
runtime's own library directories are now searched first, ahead of the
ambient `LD_LIBRARY_PATH`, for both comgr and HIP, since `rocm serve`
prepends them for both libraries equally. The loader cache and the
directory walk are now gathered once in `probe_comgr` and passed to both
the comgr and HIP searches, rather than each search reading
`LD_LIBRARY_PATH`, spawning `ldconfig -p`, and walking every managed
runtime's directories again on every invocation -- neither depends on
which library is being searched for, so the duplication cost an extra
subprocess and directory walk on every `rocm examine`/`rocm diagnose`
for no reason. `multi_copy_note` is split out as its own pure function
so the "N copies ... would load" note stays independently testable now
that `find_library_copies` only returns copies.

Two doc-comment splices, both from a missing blank `///` line between two
unrelated items: `check_18_comgr_conflict`'s doc paragraph had absorbed
check_17's above it, leaving check_17 undocumented; the same shape left
`known_install_roots` undocumented under a paragraph describing
`dedup_roots_keeping_first`. Both are split back onto the function they
actually describe.

`docs/wsl.md` said `fix-19-shm-too-small` was the only non-WSL entry that
applies on WSL; registering `check_18_comgr_conflict` for `["linux",
"wsl"]` made that false the moment this entry existed. Updated to name
both.

The "Neither option is recommended..." sentence was stated independently
in `fix.rs`'s static recipe and `diagnose.rs`'s dynamically-generated one,
in slightly different words -- already drifted, and only the `fix.rs`
copy was pinned by an e2e assertion, so the `diagnose.rs` copy could drift
further with nothing to catch it. Both now read
`fix::COMGR_CONFLICT_NEITHER_OPTION_RECOMMENDED`, so they cannot
disagree.

Not changed: `fix.rs`'s option (b) wording and `comgr_matches_runtime`'s
parity with `check_18_comgr_conflict` were flagged in an earlier review
round and were already correct by the time this round started --
verified against the current code rather than re-fixed.

Rebase onto #384 and a second symmetry gap: #384's own review fixed
`comgr_selected` picking `copies.first()` regardless of tier -- a
`rocm-install`/`managed-runtime`-only hit used to be reported as "the
one that would load" purely because it sorted first, when neither
source is one the loader consults on its own. Rebasing that fix under
this entry's HIP-runtime work surfaced a second instance of the same
asymmetry a human reviewer caught directly on this branch:
`probe_comgr`'s doc comment claimed comgr and HIP were searched and
reported identically, but only comgr ever got a `multi_copy_note` call;
the HIP branch only checked `hip.is_empty()` and pushed nothing for the
"found, but only in a directory the loader does not consult" case the
comgr side had just been fixed to report.

Fixed by stating the tier rule once and applying it to both prefixes
rather than fixing it twice: `select_loader_copy` replaces
`copies.first()` with a search restricted to `LOADER_PATH_SOURCES`
(`active-runtime`, `ld-library-path`, `loader-cache` -- the tiers the
loader itself consults), and `copies_note` replaces `multi_copy_note`
with the three-way note (no copies / more than one with a winner /
one-or-more with none selected) that a `rocm-install`-only hit now
needs. `probe_comgr` calls both exactly once for comgr and once for HIP,
so there is one call site per library rather than one function's logic
duplicated by hand into the other's branch, and the doc comment now says
what the code does instead of what it used to do before #384's fix
landed.

Pinned by `a_search_dir_only_hit_is_not_selected_for_either_prefix`,
which runs the same assertions for `COMGR_LIB_PREFIX` and
`HIP_RUNTIME_LIB_PREFIX` in one test rather than two, so the rule is
proven to apply to both rather than merely written to apply to both.
The existing `loader_cache_outranks_search_dirs` and
`the_active_runtimes_own_copy_outranks_the_ambient_environment`/
`ld_library_path_outranks_loader_cache_and_search_dirs` tests were
carried through the rebase onto `find_library_copies`/
`select_loader_copy` rather than dropped, since they pin mutation-tested
tier-ordering bugs #384 already fixed once.

`comgr_search_dirs_labels_each_tier`, which tested the now-deleted
`comgr_search_dirs_in`, is removed as redundant: `install_library_dirs`'s
own tier-labelling is already covered by
`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` and
`a_library_no_installation_claims_is_attributed_to_none`.

Verified: `cargo fmt --all -- --check`, `cargo clippy --workspace
--all-targets -- -D warnings`, `cargo clippy -p e2e-cucumber --test e2e
-- -D warnings`, and `cargo test --workspace --all-targets` all pass
(one `xtask::workflow_contract` test flaked under full-suite parallel
load on a timing assertion against a real `rocm-smi` subprocess call and
passed cleanly in isolation and on a full-suite rerun; unrelated to this
change). Cross-compiled the whole workspace, tests included, for
`x86_64-pc-windows-gnu` under `-D warnings` to confirm no code is left
unix-only-reachable and therefore dead on Windows. `cargo xtask e2e --
-n skill` passes all 6 scenarios, since this round also touches
`skills/rocm-doctor/reference.md`.

Second review round, two independent reviewers filing the same four findings
(verified rather than merely accepted):

fix-18's evidence line claimed to describe "the environment as it stands
outside the CLI's managed runtimes" unconditionally, even when a selection
actually came from the active-runtime tier -- a served child's search order,
not a plain shell's. That also means the `export LD_LIBRARY_PATH=...` remedy
can be inert for exactly that case, since `lemonade_process_environment_vars`
prepends the runtime's own directories ahead of whatever the remedy exports.
Rewording the evidence line is the easy half; the detection logic that
decides which sentence to print cannot move to plain-shell-only without
reintroducing the false positives/negatives `comgr_matches_runtime` and
`a_healthy_managed_installation_raises_no_report` were written to rule out,
since that guarantee depends on both selections already being
active-runtime-tier-aware. So `comgr_matches_runtime` is untouched;
`check_18_comgr_conflict` now checks whether either selected copy's source is
`active-runtime` and, when it is, swaps in an honest sentence plus a new
remedy note stating the export may not reach a served child and pointing at
reinstalling/repairing that runtime instead. Pinned by
`an_active_runtime_s_own_hip_without_its_own_comgr_notes_the_scope_limit`,
mutation-tested by forcing the branch condition false and watching the new
test fail on the stale sentence, then reverting.

`assert_hip_selection_is_stated`, the e2e HIP twin of the comgr selection
assertion, had only two branches, so "copies found, but none on a tier the
loader consults" -- `select_loader_copy` returning `None` on purpose, already
pinned for `libamdhip64` by
`a_search_dir_only_hit_is_not_selected_for_either_prefix` -- fell into the
`else` arm and asserted against a selected copy that does not exist. Added
the missing third branch, mirroring `assert_comgr_selection_is_stated`
exactly, including its `LOADER_PATH_SOURCES` check on the selected copy's
source, which the HIP step's final arm was also missing. Hoisted the shared
`LOADER_PATH_SOURCES` const the two steps now both use so they cannot drift
apart again.

`skills/rocm-doctor/reference.md`'s "Closed catalog (24 failure modes)"
heading and `SKILL.md`'s "the other 20 are print-only" line had both gone
stale against the table's 25 rows and 4 auto-applicable fixes
(`exactly_the_four_known_fixes_are_auto`); fixed to 25 and 21.
`rocm_doctor_skill.feature` deliberately leaves both counts unparsed against
the live CLI, and that scope decision stands -- the CLI is not wired to
report "how many rows exist" as a fact an agent reads. What it does not rule
out is checking the two numbers against each other, so
`tests/e2e-cucumber/tests/skill_reference.rs` (plain `cargo test`, no binary
needed) gained `free_standing_catalog_counts_match_the_table`, which parses
both numbers back out of the two docs and asserts them against the table's
own row and auto-fix counts. Mutation-tested by reverting each number one at
a time and confirming the new test goes red, then restoring it.

Open question from the first round, never answered: does
`install_library_dirs` calling `collect_sdk_library_paths` unconditionally
widen what a bare-metal `/opt/rocm` root probes in a way that matters?
Traced it rather than taking the earlier answer on faith: every directory
`install_library_dirs` adds for a root carries that root's own source
(`rocm-install` or `managed-runtime`, never a loader-consulted tier), and
`find_library_copies` only records a copy under that source when no
loader-consulted tier (`active-runtime`/`ld-library-path`/`loader-cache`)
already found the same path first -- so a wider probe can only add more
`rocm-install`-sourced evidence, never promote a path onto a tier
`select_loader_copy` would pick. The one place a wider probe does change an
answer is `comgr_matches_runtime`'s "does the runtime ship its own copy"
check, and there a new hit can only turn a false conflict report into `None`
(not enough evidence), which is strictly more correct, not a new risk. The
reviewer's reasoning holds; recorded here since the first round left it
unanswered.

Re-verified after this round's changes: `cargo fmt --all -- --check`, `cargo
clippy --workspace --all-targets -- -D warnings`, `cargo clippy -p
e2e-cucumber --test e2e -- -D warnings`, `cargo test --workspace
--all-targets` (251 passed, 0 failed across 33 binaries), and
`RUSTFLAGS="-D warnings" cargo check --workspace --all-targets --target
x86_64-pc-windows-gnu` all pass clean. `cargo xtask e2e -- -n skill` passes
all 6 scenarios again; `cargo xtask e2e -- -n examine-18` passes its 5 steps,
confirming the new HIP e2e branch does not break the existing scenario.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
jussielo-amd pushed a commit to jussielo-amd/rocm-cli that referenced this pull request Oct 6, 2026
… first (ROCm#384)

* feat(examine): report every code object manager library, not just the first

HIP compiles device code at run time through libamd_comgr, and a machine
can hold more than one copy. The loader picks one. When the copy it picks
does not belong to the active HIP runtime, compilation fails with an
error naming neither the library nor the second copy.

The search stopped at the first match, so the second copy could not be
seen at all. It now records every copy found, in loader search order.
Attributing each copy to the installation that owns it, and doing the
same search for the HIP runtime so the two can be compared, is deferred;
comgr_matches_runtime stays permanently unset until that lands.

The managed search now reads the runtime registry rather than listing
<data>/runtimes on disk. A directory listing yields a root and nothing
else, and the root alone does not locate a wheel runtime's libraries --
those live in a sibling _rocm_sdk_* package inside the interpreter's
site-packages, which is why `managed_sdk_ld_library_path` (the
environment the CLI builds to run an SDK tool, such as
rocm_agent_enumerator, inside a managed runtime) walks that directory.
Searching the root alone therefore missed the copy this CLI installs
itself, which is the case the entry exists for.

That knowledge now lives in one place for this search:
collect_managed_runtime_library_paths is shared by the comgr search here
and by managed_sdk_ld_library_path, so that one layout is described once
there rather than twice. A third description, collect_runtime_environment_paths,
still builds the environment a served engine process gets; unifying all
three is follow-up work, not part of this search.

Listing the directory was also a second answer to "which runtimes exist",
where the registry is the first one, and only the registry records which
site-packages an interpreter uses.

The record is preferred but not required. A read-only probe cannot depend
on one being present and current, and a host whose registry is missing or
stale is exactly the kind this command is called on: reporting no copies
there would read as a machine with nothing wrong rather than one we
failed to inspect. With no record, root's own `_rocm_sdk_*` name is read
to find its parent -- the site-packages directory its siblings share --
and a miss costs a directory that is simply not reported.

No dlopen. Reading the version through amd_comgr_get_version would run an
unknown library's initialisers on a machine called on precisely because
something is already wrong, and load a possibly conflicting HIP stack
permanently into the process. The version is read from the versioned
soname instead, and a name carrying none yields an empty version rather
than a confident wrong one.

Directory matches are now sorted, which makes the HIP probe's tie-break
deterministic where one directory holds several matching files. It used
to take whatever read_dir yielded first. Better, but a change, not a
no-op.

The loader-cache tier used to gate on which("ldconfig"), which only walks
$PATH -- and ldconfig lives in /sbin, off a non-root user's PATH on Debian
and derivatives. That silently dropped the one tier that finds a copy
registered only in ld.so.cache; it now reuses the fallback search
ldconfig_cache() already does for the same reason.

Covered at two levels, because neither is sufficient alone. Unit tests
build a wheel-format layout directly, which proves the search understands
a layout we described; only a real managed runtime proves it matches the
one the installer produces, so examine-19 asserts that on a GPU lane.

The four new fields are `serde(default)`. This structure is read back
from another machine: `rocm remote doctor` deserializes an examination
the remote's own CLI produced, and that CLI may predate these fields.
Without a default, adding one here refuses every remote running an older
build, reported as "the remote CLI is probably a different version" --
true, and useless, since the older CLI is the one that cannot be changed.

Fix CI: managed_runtime_roots resolved the data directory through
crate::runtime::default_data_dir (`$HOME/.rocm` or the OS default)
instead of AppPaths::discover, so it disagreed with the rest of the CLI
and found nothing at all on any host where ROCM_CLI_DATA_DIR relocates
the data directory -- which is exactly what every GPU e2e lane sets for
isolation. That is why the new GPU scenario
(examine-finds-the-managed-runtimes-own-compilation-library) failed
identically on every GPU family; it now resolves through AppPaths::discover
like everything else in the CLI. Also fixes a needless-collect clippy
lint in the e2e step, and tags that scenario @requires-os:linux: the
search looks only for libamd_comgr by its ELF soname, which native
Windows does not ship.

Rebasing onto main picked up two unrelated scenarios that claimed the same
numbers in turn -- examine-16 (EAI-8950, ROCm#444), then examine-17 (EAI-8449,
ROCm#434) -- so this commit's two new scenarios are renumbered to examine-18
and examine-19 to keep every scenario number in the file unique.

Review: examine-18 (the host-independent code-object-manager scenario) was
untagged and its empty-result assertion matched `Examination::default()`
alone, so it would pass with `probe_comgr` deleted outright -- on native
Windows, where the probe never runs, that is not a hypothetical. Tagged
@requires-os:linux and the assertion now also requires the "no libamd_comgr
found" note that only a probe which actually ran and came up empty pushes.
Verified the new assertion fails by temporarily suppressing that note and
watching the scenario go red, then restored it.

Review: four more issues. (1) comgr_selected and the multi-copy note named
the ld.so.cache system copy as "would load" whenever a managed wheel
runtime was active but this process's own LD_LIBRARY_PATH was empty --
wrong, because rocm serve/rocm chat prepend the active runtime's own
library directories onto LD_LIBRARY_PATH before launching the engine, so
the wheel copy is what a served process actually loads. probe_comgr now
takes the active runtime's interpreter (the same FrameworkInterpreter
already threaded through probe_framework) and searches its library_paths
first, ahead of the ambient environment; pinned by
the_active_runtimes_own_copy_outranks_the_ambient_environment. (2) The
site_packages: None fallback in collect_managed_runtime_library_paths
guessed a venv layout (root/lib/<python>/site-packages) that the installer
never produces -- root is one of the runtime's own _rocm_sdk_* package
directories, not a venv root, so the guess always found nothing. It now
reads root.parent() instead, which is the site-packages directory the
guessed layout could never reach. The test fixture
(wheel_runtime_on_disk) was built from the same wrong venv shape and so
could not have caught this; rebuilt it from the real installer layout
(ROCM_SDK_PROBE_SCRIPT's root_path), and confirmed
a_wheel_runtime_with_no_registry_record_is_still_found fails against the
old fallback before passing against the fix. (3) probe_comgr,
comgr_paths_in_loader_cache, and comgr_search_dirs had no tests at all, so
the source ordering, the ldconfig -p parsing, the /opt/rocm* scan, the
selection, and the multi-copy note could all break with the rest of the
suite still green. Split the ordering/selection logic into a pure
find_comgr_copies, the ldconfig -p line parsing into
parse_ldconfig_cache_paths, and the /opt scan into rocm_install_siblings
(opt directory taken as a parameter rather than hardcoded, so a test can
point it at a temp folder instead of the real /opt on the machine running
the suite); each is unit-tested directly, and each new test was confirmed
to fail against a deliberately broken version of the function it covers
before being restored. (4) This message previously claimed a HIP copy
list and a single shared description of the loader path launched
processes use; neither matches this diff. Corrected above.

Review: a second round, one automated and one human, reaching the same
place by different routes.

The human reviewer confirmed, independently, that the site_packages:
None fallback added in the prior round could not actually fire:
ROCM_SDK_PROBE_SCRIPT has recorded site_packages unconditionally, outside
its try, since the probe's initial version, so every manifest that
parses at all carries Some there. The test built to cover it could not
have caught that either way, since it called the function directly with
None rather than through a real candidate. Rather than keep a reachable-
looking branch and a comment describing behaviour the code does not
have, the None arm is now a no-op (root's own directories, already
collected unconditionally above, are what a hand-constructed None caller
still gets) and the test is removed. The neighbouring root-format test
is kept and re-targeted: it already called the same function directly
with None, so it is what actually exercises that arm's contract, not the
removed test.

The automated reviewer caught a real regression the first round
introduced: three new unit tests feed a path into the `:`-split
LD_LIBRARY_PATH parser without a unix cfg gate, same as the pattern the
file already uses for that reason, and that broke the required
windows-build-and-test lane. Gated, along with the helper they share
(which would otherwise be unused, and clippy-denied, once its only
callers are gated).

It also found that the active-runtime-priority fix regresses
examine-finds-the-managed-runtimes-own-compilation-library on a healthy
host: once a managed runtime is active, its directory is found first and
labelled active-runtime, and the dedup this search has always done on
resolved path then drops the later managed-runtime hit for the identical
file -- the scenario's step asserted on the label that fix makes
unreachable. The step now accepts either label as proof the CLI's own
installed copy was found, and a new unit test
(an_active_runtime_directory_that_is_also_a_managed_root_keeps_one_label)
pins that an overlapping directory keeps exactly one label rather than
being reported twice; confirmed to fail when the active-runtime search is
moved after the others, then restored.

Review: a third round, three issues.

(1) comgr_matches_runtime was declared, defaulted, and listed in the
expected-keys test, but nothing in this diff computes it -- comparing
comgr's copies against the HIP runtime's is still deferred, same as the
commit message already said. A reader of the JSON contract alone had no
way to know the field was permanently unset rather than merely not yet
exercised by a test. Dropped from this commit entirely (the struct field,
its Default entry, its expected-keys assertion); `serde(default)` makes
re-adding it free once something actually computes it.

(2) comgr_selected named the first entry in comgr_paths as "would load"
regardless of which tier found it, so a machine carrying only a leftover
/opt/rocm-* install or an inactive managed runtime -- never a copy on
active-runtime, ld-library-path, or loader-cache -- was told that stale
copy would load, when the loader does not search /opt or a managed
runtime's directory on its own. find_comgr_copies now selects only from
those three loader-consulted tiers (LOADER_PATH_SOURCES), leaving
comgr_selected None and rewording the note when every hit is
rocm-install/managed-runtime-only; pinned by
a_search_dir_only_hit_is_not_reported_as_the_one_that_would_load. The e2e
step (assert_comgr_selection_is_stated) now asserts the same tier rule
against the real JSON: selected absent plus only non-loader-tier copies
is the pass case, not a shape to tolerate.

(3) comgr_search_dirs and managed_runtime_roots had no unit tests of
their own, and swapping the loader-cache and search-dir loops inside
find_comgr_copies left every then-existing test green -- nothing pinned
their relative order. Split comgr_search_dirs into a pure
comgr_search_dirs_in (rocm_path, the /opt directory, and the resolved
managed runtimes all taken as parameters) and added
comgr_search_dirs_labels_each_tier, covering all three source labels
against temp directories. Added loader_cache_outranks_search_dirs, built
so an ld-library-path hit cannot mask the order under test. Confirmed the
claim directly: swapped the two loops, reran the suite -- exactly
loader_cache_outranks_search_dirs failed, the other 65 tests (including
ld_library_path_outranks_loader_cache_and_search_dirs) stayed green --
then reverted the swap and reran clean before committing.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>

* feat(diagnose): report a code object manager that is not the runtime's own (ROCm#385)

Builds on the copy search: with every copy of libamd_comgr known, the
catalog can say when the one that loads belongs to a different
installation than the HIP runtime that loads, and device code
compilation therefore fails with an error naming neither.

The rule is symmetric, and that is what makes it safe. The design note
for this entry proposed a special case -- "when the active runtime is
the managed runtime, the matching copy is the wheel copy" -- without
which it "fires on every healthy CLI installation". That is a patch over
a rule stated asymmetrically. Asking one question of both libraries
instead, does the libamd_comgr that would load come from the same
installation as the libamdhip64 that would load, makes every case fall
out of the rule: a healthy managed install takes both from the managed
runtime, so nothing differs and nothing is reported. A control that has
to be written as an exception is a rule that has not been stated
correctly yet.

That turns on attributing a copy to its installation correctly. A
managed runtime spreads its libraries across separate _rocm_sdk_*
packages that sit beside each other under one site-packages, not nested
under the runtime's own root at all, so ownership is decided by looking a
found file's directory up against the same directories
install_library_dirs already recorded for each root -- never by walking
up from the file, which would call each package its own install and fire
on the most common install we ship. That is the property the mutation
check targets.

Two further guards on firing: an unattributed copy is a gap in what the
search knows rather than a finding about the machine, and the runtime
must actually ship a copy of its own, or there is nothing to point the
user at.

The entry is print-only and ranks neither remedy. Removing a stack and
reordering the search path can each break a working Python environment,
and which is right depends on which stack the user means to keep.

The finding says out loud that it describes the environment outside the
managed runtimes. `rocm serve` puts a managed runtime's libraries first
on purpose and gets a different, correct answer, and a report that did
not name the environment it examined would read as a claim about one it
never looked at.

The rule above is now stated exactly once, as `comgr_matches_runtime` in
examine.rs, and both this entry and `rocm examine --json`'s field of the
same name call it rather than each restating the conditions -- so the
two surfaces cannot disagree about one machine, and a regression test
pins that. Fix option (b)'s advice matched an earlier, asymmetric
statement of the rule; it now says to prefer the wheel's own copy of
both libraries, which is what the symmetric rule actually asks the user
to do. The machine-readable inspection now carries the same
`install_root` attribution for the HIP runtime side that it already
carried for the code object manager side, since the conflict question is
about both libraries and a reader could not previously check either the
attribution or the selection for one of them.

Known roots are deduplicated across the whole list, not only where
repeats happen to land next to each other. `dedup_by` drops consecutive
duplicates, and these roots arrive from three independent sources -- the
active install, a sorted /opt scan, and the managed runtimes -- so the
active install colliding with a sibling was caught or missed depending on
where the path sorted. A root recorded twice can make a machine holding
one stack read as a machine holding two, which is the false conflict this
entry exists not to report.

hip_paths and hip_selected are `serde(default)` for the same reason the
comgr fields are: an examination read back over the remote path may have
been produced by an older CLI that never wrote them.

Once a managed runtime is found, its root resolves to its
`_rocm_sdk_devel` package directory, and the search used to guess a venv
layout (`root/lib/<python>/site-packages`) to find its sibling packages;
that guess does not hold for a real install, so the search now reads
back the library directories the SDK probe already recorded
(`library_paths`) instead of re-deriving them. This generalises the
walker introduced in ROCm#384 (`find_library_copies`/`install_library_dirs`)
so it serves the HIP side too, which is why `managed_runtime_roots` now
returns `library_paths` plural rather than a single `site_packages`
guess.

The AppPaths::discover data-dir fix and the @requires-os:linux platform
tags this commit originally carried for examine-17/18 belong with the
scenario they correct, which ROCm#384 introduced -- moved there so the
parent doesn't depend on the child to pass its own CI, and this commit
now only rebases on top of that fix rather than restating it.

Sweeping the same gap once it was visible: examine-17 (the
host-independent code-object-manager scenario, renumbered by ROCm#384's
rebase) asserts `hip_paths`/`hip_selected` the same way it asserts
`comgr_paths`/`comgr_selected`, and the empty branch had the identical
problem ROCm#384's review flagged for the comgr side -- `hip_paths: []` /
`hip_selected: null` also hold by nothing more than `Examination`'s own
defaults, so the assertion could not tell "probed, found none" from
"never probed". `probe_comgr` now pushes a "no libamdhip64 found" note
the same way it already did for libamd_comgr, and the HIP assertion now
requires it in the empty branch. Verified by temporarily suppressing the
note and watching the scenario go red, then restoring it.

Review: the attribution half still did not survive a realistic managed
runtime. `known_install_roots` registers a managed runtime by
`candidate.root_path` alone -- `_rocm_sdk_devel` when the `devel` extra
is installed -- and `libamd_comgr`/`libamdhip64` live in the sibling
`_rocm_sdk_core` package, which sits *beside* `root_path`, not under it.
A string-prefix match against the bare root therefore attributed nothing
on exactly this shape: both libraries came back `install_root: ""`,
`comgr_matches_runtime` hit its empty-root guard and returned `None`
instead of `true`, and `check_18_comgr_conflict` could not fire for that
runtime at all. The two unit tests meant to catch this
(`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime`,
`a_healthy_managed_installation_raises_no_report`) built their fixtures
from a venv root (`/data/runtimes/therock/default`,
`.../lib/python3.12/site-packages/_rocm_sdk_core/...`) that production
never produces, so they passed regardless.

Fixed at the root: `install_library_dirs` already walks, for every known
root, exactly the directories that root's copies live in (that is the
discovery-side fix this commit already made). `find_library_copies` now
builds a directory -> owning-root map from that same walk and attributes
each found file by looking its directory up in it, rather than by
string-prefix match against the bare root. A sibling package's directory
is simply one more entry in the map, keyed to the runtime that recorded
it, so it attributes correctly without being nested under anything.
`owning_install_root` and `record_library_copy` take that map instead of
the bare root list; `the_nearest_enclosing_installation_claims_a_library`
is removed because the "longest prefix wins" ambiguity it guarded against
cannot arise under exact directory lookup. Both fixture tests above were
rebuilt from the real installer shape (sibling `_rocm_sdk_*` packages
under one `site-packages`, matching `apps/rocm/src/therock.rs`'s
`ROCM_SDK_PROBE_SCRIPT`), and
`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` was
confirmed to fail against the old string-prefix attribution before
passing against the fix.

`find_library_copies` also gained the `active_runtime_dirs` parameter
runtime's own library directories are now searched first, ahead of the
ambient `LD_LIBRARY_PATH`, for both comgr and HIP, since `rocm serve`
prepends them for both libraries equally. The loader cache and the
directory walk are now gathered once in `probe_comgr` and passed to both
the comgr and HIP searches, rather than each search reading
`LD_LIBRARY_PATH`, spawning `ldconfig -p`, and walking every managed
runtime's directories again on every invocation -- neither depends on
which library is being searched for, so the duplication cost an extra
subprocess and directory walk on every `rocm examine`/`rocm diagnose`
for no reason. `multi_copy_note` is split out as its own pure function
so the "N copies ... would load" note stays independently testable now
that `find_library_copies` only returns copies.

Two doc-comment splices, both from a missing blank `///` line between two
unrelated items: `check_18_comgr_conflict`'s doc paragraph had absorbed
check_17's above it, leaving check_17 undocumented; the same shape left
`known_install_roots` undocumented under a paragraph describing
`dedup_roots_keeping_first`. Both are split back onto the function they
actually describe.

`docs/wsl.md` said `fix-19-shm-too-small` was the only non-WSL entry that
applies on WSL; registering `check_18_comgr_conflict` for `["linux",
"wsl"]` made that false the moment this entry existed. Updated to name
both.

The "Neither option is recommended..." sentence was stated independently
in `fix.rs`'s static recipe and `diagnose.rs`'s dynamically-generated one,
in slightly different words -- already drifted, and only the `fix.rs`
copy was pinned by an e2e assertion, so the `diagnose.rs` copy could drift
further with nothing to catch it. Both now read
`fix::COMGR_CONFLICT_NEITHER_OPTION_RECOMMENDED`, so they cannot
disagree.

Not changed: `fix.rs`'s option (b) wording and `comgr_matches_runtime`'s
parity with `check_18_comgr_conflict` were flagged in an earlier review
round and were already correct by the time this round started --
verified against the current code rather than re-fixed.

Rebase onto ROCm#384 and a second symmetry gap: ROCm#384's own review fixed
`comgr_selected` picking `copies.first()` regardless of tier -- a
`rocm-install`/`managed-runtime`-only hit used to be reported as "the
one that would load" purely because it sorted first, when neither
source is one the loader consults on its own. Rebasing that fix under
this entry's HIP-runtime work surfaced a second instance of the same
asymmetry a human reviewer caught directly on this branch:
`probe_comgr`'s doc comment claimed comgr and HIP were searched and
reported identically, but only comgr ever got a `multi_copy_note` call;
the HIP branch only checked `hip.is_empty()` and pushed nothing for the
"found, but only in a directory the loader does not consult" case the
comgr side had just been fixed to report.

Fixed by stating the tier rule once and applying it to both prefixes
rather than fixing it twice: `select_loader_copy` replaces
`copies.first()` with a search restricted to `LOADER_PATH_SOURCES`
(`active-runtime`, `ld-library-path`, `loader-cache` -- the tiers the
loader itself consults), and `copies_note` replaces `multi_copy_note`
with the three-way note (no copies / more than one with a winner /
one-or-more with none selected) that a `rocm-install`-only hit now
needs. `probe_comgr` calls both exactly once for comgr and once for HIP,
so there is one call site per library rather than one function's logic
duplicated by hand into the other's branch, and the doc comment now says
what the code does instead of what it used to do before ROCm#384's fix
landed.

Pinned by `a_search_dir_only_hit_is_not_selected_for_either_prefix`,
which runs the same assertions for `COMGR_LIB_PREFIX` and
`HIP_RUNTIME_LIB_PREFIX` in one test rather than two, so the rule is
proven to apply to both rather than merely written to apply to both.
The existing `loader_cache_outranks_search_dirs` and
`the_active_runtimes_own_copy_outranks_the_ambient_environment`/
`ld_library_path_outranks_loader_cache_and_search_dirs` tests were
carried through the rebase onto `find_library_copies`/
`select_loader_copy` rather than dropped, since they pin mutation-tested
tier-ordering bugs ROCm#384 already fixed once.

`comgr_search_dirs_labels_each_tier`, which tested the now-deleted
`comgr_search_dirs_in`, is removed as redundant: `install_library_dirs`'s
own tier-labelling is already covered by
`a_library_deep_inside_a_managed_runtime_belongs_to_the_runtime` and
`a_library_no_installation_claims_is_attributed_to_none`.

Verified: `cargo fmt --all -- --check`, `cargo clippy --workspace
--all-targets -- -D warnings`, `cargo clippy -p e2e-cucumber --test e2e
-- -D warnings`, and `cargo test --workspace --all-targets` all pass
(one `xtask::workflow_contract` test flaked under full-suite parallel
load on a timing assertion against a real `rocm-smi` subprocess call and
passed cleanly in isolation and on a full-suite rerun; unrelated to this
change). Cross-compiled the whole workspace, tests included, for
`x86_64-pc-windows-gnu` under `-D warnings` to confirm no code is left
unix-only-reachable and therefore dead on Windows. `cargo xtask e2e --
-n skill` passes all 6 scenarios, since this round also touches
`skills/rocm-doctor/reference.md`.

Second review round, two independent reviewers filing the same four findings
(verified rather than merely accepted):

fix-18's evidence line claimed to describe "the environment as it stands
outside the CLI's managed runtimes" unconditionally, even when a selection
actually came from the active-runtime tier -- a served child's search order,
not a plain shell's. That also means the `export LD_LIBRARY_PATH=...` remedy
can be inert for exactly that case, since `lemonade_process_environment_vars`
prepends the runtime's own directories ahead of whatever the remedy exports.
Rewording the evidence line is the easy half; the detection logic that
decides which sentence to print cannot move to plain-shell-only without
reintroducing the false positives/negatives `comgr_matches_runtime` and
`a_healthy_managed_installation_raises_no_report` were written to rule out,
since that guarantee depends on both selections already being
active-runtime-tier-aware. So `comgr_matches_runtime` is untouched;
`check_18_comgr_conflict` now checks whether either selected copy's source is
`active-runtime` and, when it is, swaps in an honest sentence plus a new
remedy note stating the export may not reach a served child and pointing at
reinstalling/repairing that runtime instead. Pinned by
`an_active_runtime_s_own_hip_without_its_own_comgr_notes_the_scope_limit`,
mutation-tested by forcing the branch condition false and watching the new
test fail on the stale sentence, then reverting.

`assert_hip_selection_is_stated`, the e2e HIP twin of the comgr selection
assertion, had only two branches, so "copies found, but none on a tier the
loader consults" -- `select_loader_copy` returning `None` on purpose, already
pinned for `libamdhip64` by
`a_search_dir_only_hit_is_not_selected_for_either_prefix` -- fell into the
`else` arm and asserted against a selected copy that does not exist. Added
the missing third branch, mirroring `assert_comgr_selection_is_stated`
exactly, including its `LOADER_PATH_SOURCES` check on the selected copy's
source, which the HIP step's final arm was also missing. Hoisted the shared
`LOADER_PATH_SOURCES` const the two steps now both use so they cannot drift
apart again.

`skills/rocm-doctor/reference.md`'s "Closed catalog (24 failure modes)"
heading and `SKILL.md`'s "the other 20 are print-only" line had both gone
stale against the table's 25 rows and 4 auto-applicable fixes
(`exactly_the_four_known_fixes_are_auto`); fixed to 25 and 21.
`rocm_doctor_skill.feature` deliberately leaves both counts unparsed against
the live CLI, and that scope decision stands -- the CLI is not wired to
report "how many rows exist" as a fact an agent reads. What it does not rule
out is checking the two numbers against each other, so
`tests/e2e-cucumber/tests/skill_reference.rs` (plain `cargo test`, no binary
needed) gained `free_standing_catalog_counts_match_the_table`, which parses
both numbers back out of the two docs and asserts them against the table's
own row and auto-fix counts. Mutation-tested by reverting each number one at
a time and confirming the new test goes red, then restoring it.

Open question from the first round, never answered: does
`install_library_dirs` calling `collect_sdk_library_paths` unconditionally
widen what a bare-metal `/opt/rocm` root probes in a way that matters?
Traced it rather than taking the earlier answer on faith: every directory
`install_library_dirs` adds for a root carries that root's own source
(`rocm-install` or `managed-runtime`, never a loader-consulted tier), and
`find_library_copies` only records a copy under that source when no
loader-consulted tier (`active-runtime`/`ld-library-path`/`loader-cache`)
already found the same path first -- so a wider probe can only add more
`rocm-install`-sourced evidence, never promote a path onto a tier
`select_loader_copy` would pick. The one place a wider probe does change an
answer is `comgr_matches_runtime`'s "does the runtime ship its own copy"
check, and there a new hit can only turn a false conflict report into `None`
(not enough evidence), which is strictly more correct, not a new risk. The
reviewer's reasoning holds; recorded here since the first round left it
unanswered.

Re-verified after this round's changes: `cargo fmt --all -- --check`, `cargo
clippy --workspace --all-targets -- -D warnings`, `cargo clippy -p
e2e-cucumber --test e2e -- -D warnings`, `cargo test --workspace
--all-targets` (251 passed, 0 failed across 33 binaries), and
`RUSTFLAGS="-D warnings" cargo check --workspace --all-targets --target
x86_64-pc-windows-gnu` all pass clean. `cargo xtask e2e -- -n skill` passes
all 6 scenarios again; `cargo xtask e2e -- -n examine-18` passes its 5 steps,
confirming the new HIP e2e branch does not break the existing scenario.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>

---------

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-hub-reviewed agent-hub has reviewed this

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants