Repository navigation
fix(runtimes): resolve a runtime selector to the key that matches exactly - #497
Conversation
…ctly `storage remove-old-installs` decides about a manifest and then names it to the deleter by key -- and the two sides resolved that key differently. `build_prune_plan` matched `manifest.runtime_key == runtime_key`, while `select_runtime_manifest` matched `eq_ignore_ascii_case` and took the first hit from a newest-first list. With two registry keys differing only in letter case, prune decided about one install and acted on the other. That pair is reachable without anything exotic: generated keys are slugified to lower case, while `runtimes adopt --runtime-key` and `runtimes import` store the key verbatim, without slugifying it. Both record the result as read-only, and adopt stamps `installed_at_unix_ms` with the moment it ran, so the adopted record is the newest and won the case-insensitive lookup. The registry's duplicate guard compares file paths, and `ABC.json` and `abc.json` are two files on a case-sensitive filesystem, so nothing rejects the pair. The effect was not a deleted folder -- an adopted record is read-only, so its install root is left alone -- but the adopted runtime was silently de-registered, the install the user confirmed stayed, and the command reclaimed nothing while naming two different runtimes in one sentence. Fixed in the resolver rather than by handing the deleter a manifest, because several callers resolve the same key more than once and must land on the same record every time: the uninstall revalidation re-derives its plan from disk after a confirmation prompt that can block indefinitely, and activate, rollback, update and engine selection all round-trip a key. Passing a manifest would have fixed prune alone and left the rest. An exact key match is now taken first and wins outright. The case-insensitive fallback -- a deliberate convenience, so a key typed in the wrong case still resolves -- applies only when it lands on a single record. Two case twins are reported as the ambiguity they are, listing both keys, the same way a `runtime_id` naming several installs already is. That error is unreachable for a user whose keys do not differ only in case: two all-lowercase keys that match case-insensitively are byte-equal, and the registry holds one file per key. Both regression tests fail before this change. The filesystem one needs a case-sensitive filesystem to hold two such registry entries at all, so it is ignored elsewhere; the in-memory one runs on every platform, including the Windows lane, and is the one that goes red on revert. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
jussielo-amd
left a comment
There was a problem hiding this comment.
Summary
Bug fix — select_runtime_manifest now tries an exact runtime_key match before falling back to case-insensitive matching (erroring on case-ambiguity). Fixes a silent-wrong-deletion bug in storage remove-old-installs when two installed runtimes have keys differing only in case.
Narrowly scoped and unusually well-verified — reverting the resolver locally reproduces the exact reported bug, and the fix restores correct behavior. cargo clippy/cargo fmt --check clean. No dead code, stubs, or bugs found.
Non-blocking suggestions
- The Gherkin-exemption justification covers
remove-old-installs/adopt/import, but the new ambiguity error is also newly surfaced viaruntimes activate,uninstall, andupdate --runtime— user-typed selectors, not just internal round-trips. Worth tightening per AGENTS.md §3. - The PR description states no Gherkin scenario exists anywhere for the affected commands — scenarios do exist for
adopt(runtime_setup.feature:140-145) andimport(runtime_lifecycle.feature:42-50), just not covering this specific case-ambiguity behavior. Worth correcting the wording; the substantive point (no existing coverage of this bug) still holds. docs/testing.md/docs/manual-testing.mdare untouched — a judgment call given how narrow the edge case is, but AGENTS.md §5 asks for these when observable behavior changes.
Positive signals
- Verified empirically, not just read: revert-and-rerun reproduced the exact reported bug, then passed cleanly once restored.
- Added both a fast cross-platform unit test and a filesystem integration test (correctly
ignore-gated to Linux with a confirmed-necessary reason). - Reuses
runtime_keys_textrather than duplicating, and simplifies a previous inline.map().join(). - PR description is unusually precise about why the fix belongs in the shared resolver (7 call sites), and proactively discloses a known related out-of-scope bug.
Tradeoff (noted, not a concern)
Fix patches resolution (read time) rather than rejecting case-twin keys at creation (write time) — explicitly and reasonably justified in the PR description as avoiding new failure modes in adopt/import while still fixing registries that already hold such a pair.
|
The non-blocking items from the approving review, addressed after merge:
|
…#529) * test(runtimes): cover the case-twin selector refusal end to end ROCm#497 made a runtime selector that matches two installed keys only by letter case fail with an error naming both, instead of silently acting on one. That is user-observable output, so per AGENTS.md section 3 it gets a scenario, not only the unit and filesystem tests that shipped with the fix. runtime-lifecycle-09 plants two runtimes whose keys differ only in case (Linux-only: they are two registry files, which can only coexist on a case-sensitive filesystem), activates with a selector matching both only by case, and asserts the refusal names both keys AND that nothing became active. Then it follows the advice: naming one key exactly activates that runtime and not its twin. Mutation-checked: with the resolver back to silently taking the first match, the scenario fails at the refusal step. Also documents the behaviour in docs/testing.md and docs/manual-testing.md, which ROCm#497's review asked for. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(runtimes): prove the case twins are two files before relying on it runtime-lifecycle-09's premise is that two keys differing only in case are two registry files, and `@requires-os:linux` was standing in for that. It is not the same thing: the isolated root follows `$TMPDIR`, which can be a casefolded or mounted case-insensitive directory on a Linux host. There the second plant overwrites the first, the selector has a single case-insensitive match, activation succeeds, and the scenario failed at the refusal step blaming the resolver for a fixture that never existed. The Given step now counts the registry entries and fails, naming the filesystem, when the twins collapsed. Counting is what works: checking for the lower-case file would pass on such a filesystem, because that path resolves to the upper-case one. Checked by planting the same file twice: the scenario now fails at the Given with that message instead of at the refusal. Also stop enumerating commands in docs/testing.md. The refusal is a property of how any runtime selector is resolved, and the list of three already missed `engines install --runtime-id`; say that every command taking a runtime selector refuses instead. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(testing): scope the case-twin refusal to the shared resolver The paragraph said every command taking a runtime selector refuses a case-only match. Two do not: ComfyUI's installer keeps its own lookup and takes the first case-insensitive match, and the config set-default-runtime and set-engine --runtime-id commands store a selector without resolving it. Say which commands refuse and name the exceptions. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(e2e): skip the case-twin scenario where the temp root folds case The case-twin selector scenario needs two registry files whose names differ only in letter case. @requires-os:linux does not guarantee that: the isolated root follows $TMPDIR, which on Linux and WSL2 can sit on a casefolded directory or a mounted Windows drive. Such a host cannot hold the premise, and this suite reports that as not-applicable, not failed. Add a capability probe that writes two case-twin names under the same temp root the scenarios use and checks they stayed two files, and a @requires-case-sensitive-fs gate beside the other host gates. Tag the scenario with it. The entry count in the Given stays as a guard for runs that bypass the gates via cucumber's -n/--tags, and now says so. Share one registry_dir helper between planting and counting so the two cannot drift. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(testing): reflow the case-twin selector paragraph Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(e2e): report the OS before the case-folding reason when both gates fail On native Windows a scenario tagged both @requires-os:linux and @requires-case-sensitive-fs failed both premises, and the grid reported the filesystem reason. Check requires_os first so the coarser reason wins. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(e2e): route the remaining registry paths through registry_dir Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(e2e): list the OS gate before the case-folding gate in resolve() The doc list still gave the two gates in the order the body had before the OS check was moved ahead of the case-folding one. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> --------- Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
`rocm storage remove-old-installs` deletes multi-gigabyte install folders from a set the user never listed, so its safety rules are worth testing over generated input rather than examples alone. Property tests for the pure policy (`select_runtimes_to_remove`), 2048 cases each: - the install in use, the rollback target, anything the active-runtime marker names and anything ROCm CLI does not own are never selected, including at `--keep 0`; - each retention group keeps exactly its `--keep` most recent installs that nothing else holds (ties broken by key) and removes the rest, and raising `--keep` never removes more; - every install is kept or removed exactly once, and each kept install is reported with the first hold reason that applies; - every selected key resolves, through the real `select_runtime_manifest`, to that same install rather than to a case twin of it. The expected groups and holds are computed in the test from the `--keep` help and the matching rules the policy implements, not with the policy's own helpers. Generators draw from small, weighted alphabets so case-twin keys, shared runtime ids, tied install times, flag/specs disagreement on the toolchain choice and groups with more candidates than `--keep` actually occur; a deterministic reach report fails if they stop. A second property, 256 cases, applies whole plans to real registry files and install folders through the real removal call and checks the install in use survives. The checked-in regression seed is a case-twin input that fails when the runtime selector fix from #497 is reverted. docs/testing.md gains a "Removing Old Installs" section listing the commands. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…ew points Main gained an architecture-doc check that exempts property-test files only by a `_properties`-style stem, so the bare `storage/properties.rs` would fail it. Move it to `storage_properties.rs`, declared through `#[path]` as the same `properties` module (as `process_tree_properties.rs` is), so test paths and the documented `storage::properties` filter are unchanged, and give it an architecture row like its sibling property files. Drop the saved proptest seed instead of carrying it to the new name. It held the #497 case-twin failure as an RNG seed, which replays a different input once the generators change; that input is pinned as the literal example `a_case_twin_pair_resolves_to_the_install_the_policy_cleared`, so the seed was redundant, and left in place it would also be orphaned by the move. Read both suites' case counts through `proptest_cases`, so `PROPTEST_CASES` can raise them as it does for the other suites. Count the four single-install toolchain rows of `generator_reach_report` in groups of one too: a lone install whose toolchain is misread can still move into a sibling group. Floors re-measured at four fifths of the new counts. Correct what the resolver property is said to guard: generated case twins do fail it when the resolver tries a case-insensitive match first (checked by reverting that precedence), so it is not only the pinned example that covers #497. State why the hold-reason oracle follows the policy's own order, and say which tests the `storage::properties` filter runs besides properties. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. — no rows reference this area.docs/architecture.md. — N/A.Fixes #496
Summary
rocm storage remove-old-installsdecides which install to remove by inspecting a manifest, then names it to the deleter by key string — and the two sides resolved that key differently. The planner matched exactly;select_runtime_manifestmatched case-insensitively and took the first hit from a newest-first list. With two registry keys differing only in letter case, prune decided about one install and acted on the other.That pair is reachable without anything exotic: generated keys are slugified to lower case, while
runtimes adopt --runtime-keytakes the key the user typed andruntimes importtakes it from the supplied manifest — neither slugifies. Both record the result read-only, and adopt stampsinstalled_at_unix_mswith the moment it ran, so the adopted record is newest and won the lookup. The registry's duplicate guard compares file paths, andABC.jsonandabc.jsonare two files on a case-sensitive filesystem.The effect is not a deleted folder — an adopted record is read-only, so its install root is left alone — but the adopted runtime is silently de-registered, the install the user confirmed stays, and the command reclaims nothing while naming two different runtimes in one sentence.
Risk: low-medium. It changes a resolver shared by seven call sites, but behaviour differs only when two installed keys differ solely in case. Every caller was enumerated; those that pass a stored key (
rollback, the uninstall revalidation, prune itself, engine env selection) always hit the exact branch.Why the resolver, and not the deleter
Handing the deleter the manifest the planner already resolved looks like the smaller fix, and it was rejected: several callers resolve the same key more than once and must land on the same record every time. The uninstall revalidation re-derives its plan from disk after a confirmation prompt that can block indefinitely — deliberately, so a concurrent change cannot be applied against stale data — and activate, rollback, update and engine selection all round-trip a key. Fixing the deleter would have fixed prune alone and left the identical mismatch in the revalidation path.
What the new behaviour is
An exact key match is taken first and wins outright. The case-insensitive fallback — a deliberate convenience, so a key typed in the wrong case still resolves — applies only when it lands on a single record. Two case twins are reported as the ambiguity they are, listing both keys, mirroring the existing error for a
runtime_idthat names several installs.That error is unreachable for a user whose keys do not differ only in case: two all-lowercase keys that match case-insensitively are byte-equal, and the registry holds one file per key. The user who does see it previously got a silent, order-dependent pick between two different installs — for
activateanduninstall, an action on the wrong runtime.Test plan
Both regression tests fail before this change and pass after; verified by reverting only the resolver body.
runtime_selector_prefers_the_exact_key_over_a_case_twin— builds the twin in memory, so it runs on every platform including the Windows lane, and is the one that goes red on revert.prune_removes_the_install_it_named_and_not_a_case_twin_of_it— drives real registry files, real install folders and the real removal call, and additionally asserts the adopted record still resolves as read-only so no folder becomes deletable. Ignored off Linux: two entries differing only in case cannot be staged on a case-insensitive filesystem, where the second fixture would overwrite the first.cargo test --workspace --all-targets3260 passed / 0 failed (exit 0);cargo clippy --locked --workspace --all-targets -- -D warnings,cargo fmt --all --check,cargo xtask manifest --check,cargo xtask tpn --checkall exit 0.On the Gherkin requirement
No scenario is added, and that is a gap rather than an exemption. Scenarios do exist for these commands —
runtimes adopt(@id:runtime-adopt-preexisting-rejected, runtime-07) andruntimes import, activate, rollback and uninstall (runtime_lifecycle.feature, e.g. runtime-lifecycle-06) — but none covers two installed runtimes whose keys differ only in case, which is what this fix is about. An earlier version of this paragraph said no scenario existed for these commands at all; that was wrong.The new ambiguity error is also reachable from user-typed selectors, not only internal round-trips:
runtimes activate,runtimes uninstallandupdate --runtimeall resolve throughselect_runtime_manifest, so a user with a case-twin pair in their registry now gets an explicit error there instead of a silent wrong pick. Covering that end to end needs a multi-runtime install fixture; today the behaviour is covered by a unit test and a filesystem-level test driving the real registry and the real removal path.Follow-ups deliberately not in this PR
adopt/importwith its own questions.rocm runtimes uninstallon one twin still clears the other twin's active pointer and deletes its activation marker, because those comparisons are case-insensitive. That loses state for a runtime still installed. Prune cannot reach it, because the retention holds keep both twins.adopt --runtime-key " abc "stores a key no selector can ever resolve, since the selector is trimmed first.