Repository navigation
Conversation
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · d5a72f1
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change (test-only: no production code touched).
No ticket named. Description references PR #497 as prior context only, not as a ticket this PR closes.
Blocking
None
Non-blocking
generator_reach_report's rows 12–15 undercount singleton retention groups —apps/rocm/src/storage/properties.rs:727-747
Rows 7–11 (tied install times, case-twin keys, channel/format/family spelled two ways) are inherently pairwise and correctly gated behindmembers.len() > 1. Rows 12–15 (develflag disagreeing with specs, spaced extra, near-miss extra, another package's spec first) are single-manifest.any()checks copied under the samelen() > 1guard, so a group of exactly one install exhibiting any of those conditions is silently excluded from the reach count. This doesn't make anyproptest!property vacuous — those run over the full generated group regardless of size — but it means the floor for rows 12–15 under-measures true generator reach, and a regression that only cost the generator its singleton-group occurrences of these conditions would not tripgenerator_reach_report.
Confidence 85 · mechanical · Fix: move the four.any()checks for rows 12–15 outside themembers.len() > 1block (or add a second, ungated block for them).
Decisions for the author
the_deleter_resolves_each_selected_key_to_the_install_the_policy_clearedis guaranteed to pass on every freshly-generated case under today's code — non-blocking-improvement
select_runtime_manifest(apps/rocm/src/runtimes_cmd.rs:1656-1670) matches onruntime_key == selectorfirst and returns immediately; the key this property passes in is always an exact clone of an existing manifest'sruntime_key(produced byselect_runtimes_to_remove), so the exact-match branch fires on every case the 2048-sample run generates — the case-insensitive fallback this property is nominally probing is never reached by fresh generation. The property's own doc comment is honest about why it still earns its place ("the real resolver is called here… so this keeps holding whatever that rule becomes") and its practical protection is real: it would catch a regression that removed or weakened the exact-match-first precedence, and the checked-in regression seed (apps/rocm/proptest-regressions/storage/properties.txt) replays the historical #497 case-twin failure on every run regardless of fresh generation. Worth confirming this is the understood tradeoff (day-to-day protection comes from the seed plusselect_runtime_manifest's own direct tests, not from this property's random cases) rather than something a future reader should assume is a live, generically-exercised cross-check.applying_a_whole_plan_never_removes_the_install_in_useonly asserts the negative direction — tradeoff
apps/rocm/src/storage.rs:1842-1877builds a plan, executes everyuninstall_runtimecall, and then only checks that the active install's folder survived — it never asserts that the entries inplan.removewere actually deleted, so a silent no-op in the removal loop (nothing ever gets removed) would still pass. This is a reasonable, deliberate scope choice: removal-actually-happened coverage already exists in the pre-existing hand-writtenprune_removes_the_install_it_named_and_not_a_case_twin_of_it(apps/rocm/src/storage.rs:1742), which asserts both that the named install's folder and registry entry are gone and that the surviving ones remain. Flagging so the asymmetry is explicit rather than assumed.
Positive signals
- Re-deriving expected groups/holds independently of the production helpers (
expected_default_key,held_regardless_of_recency,installed_the_toolchain,expected_groupinproperties.rs) rather than callingresolved_default_runtime_key/unconditional_hold/includes_devel/retention_groupdirectly is exactly the right design for a policy this safety-sensitive — traced line-by-line against production and found faithful in every case, including the subtler case-insensitive-everywhere and devel-flag/package-spec disagreement rules. generator_reach_reportas a deterministic, floor-checked measurement of generator coverage (not just the properties themselves) is an uncommon and valuable practice that catches the generators silently drifting away from the edge cases the properties depend on.- The weighted, tiny alphabets (
KEYS,CHANNELS,FORMATS,FAMILIES, theToolchainenum) are a well-reasoned, explicitly-commented answer to the standard "uniform random strings never collide" failure mode for property tests over anything key-based.
Deployment notes
None — test and documentation only, no dependency, migration, or behavior change.
What this covered
Read completely: apps/rocm/src/storage/properties.rs (new file, 782 lines), the whole of apps/rocm/src/storage.rs (1934 lines, including the new mod properties; declaration and the new AppliedPlan/apply_generated_plan/applying_a_whole_plan_never_removes_the_install_in_use test code), docs/testing.md's new section, and apps/rocm/proptest-regressions/storage/properties.txt. Targeted reads in apps/rocm/src/therock.rs (InstalledRuntimeManifest, WheelRuntimeComposition, includes_devel, wheel_composition_includes_devel, load_runtime_manifests) and apps/rocm/src/runtimes_cmd.rs (select_runtime_manifest, uninstall_runtime), plus the full root AGENTS.md, git history/blame around the touched regions, and PR #497 (the fix this suite guards) for context. Reviewed against commits 9301dfc2…d5a72f1d.
Fanned out two independent passes (storage/properties correctness; docs+history+agent-instruction adherence+impact analysis), consolidated their findings against the source myself, and ran the design-review pass directly (reading select_runtime_manifest and the pre-existing hand-written regression test myself to settle the two items above) rather than spawning further sub-workers for it, since the two code-reading passes had already surfaced the material to decide both questions.
Did not run: cargo test/cargo xtask coverage — no Rust toolchain available in this environment, so the PR's own mutation-evidence table and "31/31 passing" claim are read from CI and the PR description, not independently executed; CI's build-and-test, Test (affected crates), and clippy checks are all green for this exact commit, which is treated at face value. All CI checks for d5a72f1d are SUCCESS or SKIPPED except two self-hosted E2E lanes (E2E tests (Strix Halo, Ubuntu, release/nightly, current)), which show CANCELLED; the self-hosted consolidated report finished SUCCESS afterward and every other self-hosted hardware lane (MI300X, Strix Halo Windows/WSL2, W7900) succeeded, and this change touches no e2e-relevant code, so this reads as unrelated infra rather than a result of this PR — noted rather than assumed. History and prior-changes passes ran (PR #497 and its tests inspected directly). Not reconciled against any PR discussion/comments — none were read, by design.
jussielo-amd
left a comment
There was a problem hiding this comment.
Looks good, thanks Romain. No blockers: the oracle recomputes groups/holds without the production helpers, the reach report is sound, and the docs counts match.
Rebase note: one conflict with main, in docs/testing.md (both sides add a section before "Provider-Assisted Planning"); keep both. No code conflict; the re-exports the tests use are still there.
Non-blocking, worth a follow-up
applying_a_whole_plan_never_removes_the_install_in_useonly asserts the active install survives; it would also pass ifplan.removewere empty, and it skips the production loop's pre-delete hold re-check. Asserting that planned installs are actually gone (or extracting the production loop) would tighten it.- No direct "configured default is never selected" property: at
--keep 0, dropping only the default hold survives (the PR's mutation row drops active and default together). - The checked-in seed's "replays the case-twin" wording can't be shown from the shrunk value; the hand-written
prune_removes_the_install_it_named_and_not_a_case_twin_of_itcovers it anyway, so softening the wording is enough. - Reach row 19 (floor 130, expected ~160) has a thin margin if the proptest RNG stream ever changes.
🤖 by agenthub on AMD AgentHub
juhovainio
left a comment
There was a problem hiding this comment.
Thanks for the thorough property suite; the oracle being derived from the --keep help text rather than the policy's own helpers is a good call, and the reach report is a nice guard against vacuous generators. One blocker and a few smaller points.
Needs fixing before merge
- The branch currently conflicts with main. After rebasing, please re-run the
storage::tests, sincestorage.rs/runtimes_cmd.rsmay have moved underneath this.
Worth addressing (inline comments below)
- The persisted proptest seed is an RNG seed, not the input, so the claim that it fails when the #497 fix is reverted isn't actually shown (the mutation check was done with the seed file removed).
apply_generated_planleaks its temp dir on early?returns.- The hold-reason property can panic on a map index instead of giving a readable failure.
- The
saw[..]/labels/FLOORalignment in the reach report is by position only.
For awareness
- The 20k-sample reach report has hard-coded floors, so they need re-measuring whenever a generator changes. Deterministic, so not flaky, but a maintenance cost.
- One line in
docs/testing.mdis over-long (cosmetic). - The two Strix Halo Ubuntu E2E jobs (nightly, release) hit the 45m job limit; that looks like a runner timeout for a test-only PR, but they should be rerun after the rebase to confirm.
| # | ||
| # It is recommended to check this file in to source control so that | ||
| # everyone who runs the test benefits from these saved cases. | ||
| cc 4cab42fe7419af3e985b4da3f1ad9faff893c57b8dc633d49f6c4b87054ca7dd # shrinks to manifests = [InstalledRuntimeManifest { runtime_key: "A", runtime_id: "therock-release:gfx110X-all", channel: "release", format: "wheel", family: "gfx110X-all", family_source: "test", version: "7.0.0", install_root: "rocm-storage-proptest/A", selected_artifact_url: "https://example.invalid/therock", source_layout_generation: None, index_url: None, tarball_file_name: None, python_launcher: None, python_executable: None, pip_cache_dir: None, rocm_sdk: None, sdk_torch: None, wheel_composition: Some(WheelRuntimeComposition { source_layout_generation: "canonical", package_specs: ["rocm[libraries,devel,device-gfx1100]==7.0.0"], rocm_sdk_target: Some("gfx1100") }), read_only: false, imported_from: None, devel: true, installed_at_unix_ms: 0 }, InstalledRuntimeManifest { runtime_key: "a", runtime_id: "therock-release:gfx110X-all", channel: "release", format: "wheel", family: "gfx110X-all", family_source: "test", version: "7.0.0", install_root: "rocm-storage-proptest/a", selected_artifact_url: "https://example.invalid/therock", source_layout_generation: None, index_url: None, tarball_file_name: None, python_launcher: None, python_executable: None, pip_cache_dir: None, rocm_sdk: None, sdk_torch: None, wheel_composition: Some(WheelRuntimeComposition { source_layout_generation: "canonical", package_specs: ["rocm[libraries,devel,device-gfx1100]==7.0.0"], rocm_sdk_target: Some("gfx1100") }), read_only: false, imported_from: None, devel: true, installed_at_unix_ms: 0 }], inputs = RetentionInputs { active_runtime_key: None, previous_runtime_key: None, default_runtime_id: None, marker_runtime_keys: [] }, keep = 0 |
There was a problem hiding this comment.
This line stores an RNG seed, not the input; any change to the generators makes it replay a different case. The PR notes the #497 mutation was checked with this file removed, so please either pin the case-twin input as an explicit example test or soften the claim that this seed fails when the fix is reverted.
There was a problem hiding this comment.
Agreed: the file holds an RNG seed, not the input, so it stops replaying the case-twin pair as soon as the generators change. aa23004 pins the input instead. a_case_twin_pair_resolves_to_the_install_the_policy_cleared builds the two installs whose keys differ only in case, first asserts the policy selects both, then checks each key resolves to its own install. With the #497 fix reverted and the seed file deleted, it fails with the plan says remove "a", but the deleter resolves that key to "A". The property's doc comment and docs/testing.md now also say that generated cases only take the resolver's exact-match branch. The seed file stays since proptest replays it for free, but nothing relies on it any more.
| in_use_key: active.runtime_key.clone(), | ||
| folder_survives: active.install_root.is_dir(), | ||
| }; | ||
| let _ = std::fs::remove_dir_all(root); |
There was a problem hiding this comment.
The temp root is only removed on the success path; an early ? in apply_generated_plan leaks it. A drop guard would clean up on every path.
There was a problem hiding this comment.
Fixed in aa23004: apply_generated_plan now holds a RemoveOnDrop guard on the test root, so the root is removed on every path, including an early ?.
| Some(HoldReason::Marker) | ||
| } else if manifest.read_only || manifest.imported_from.is_some() { | ||
| Some(HoldReason::NotOwned) | ||
| } else if groups[&expected_group(manifest)] |
There was a problem hiding this comment.
groups[&expected_group(manifest)] panics with a map-index error when the oracle and the policy disagree about whether an install is held. Use .get() with a prop_assert! so the failure names the install.
There was a problem hiding this comment.
Fixed in aa23004: it now looks the group up with .get() and fails with a prop_assert! that names the install. Only installs that none of the earlier checks hold reach that branch, and those are exactly the ones recency_groups buckets, so a missing group would mean those checks and held_regardless_of_recency have drifted apart; the assertion now says so rather than panicking.
| let mut runner = TestRunner::deterministic(); | ||
| let draw = (manifests(), retention_inputs()); | ||
|
|
||
| let labels: [&str; ROWS] = [ |
There was a problem hiding this comment.
saw[0..19] is indexed by position against labels and FLOOR; reordering any one of the three silently misaligns them. Named consts or a single struct array would prevent that.
There was a problem hiding this comment.
Fixed in aa23004: each row's label and floor now live together in one REACH table of (Row, label, floor), saw is indexed by Row::… as usize, and the report first asserts every table row sits at its own Row index. Swapping two rows makes generator_reach_report fail. The measured counts are unchanged.
`rocm storage remove-old-installs` deletes multi-gigabyte install folders from a set the user never listed, so its safety rules are worth testing over generated input rather than examples alone. Property tests for the pure policy (`select_runtimes_to_remove`), 2048 cases each: - the install in use, the rollback target, anything the active-runtime marker names and anything ROCm CLI does not own are never selected, including at `--keep 0`; - each retention group keeps exactly its `--keep` most recent installs that nothing else holds (ties broken by key) and removes the rest, and raising `--keep` never removes more; - every install is kept or removed exactly once, and each kept install is reported with the first hold reason that applies; - every selected key resolves, through the real `select_runtime_manifest`, to that same install rather than to a case twin of it. The expected groups and holds are computed in the test from the `--keep` help and the matching rules the policy implements, not with the policy's own helpers. Generators draw from small, weighted alphabets so case-twin keys, shared runtime ids, tied install times, flag/specs disagreement on the toolchain choice and groups with more candidates than `--keep` actually occur; a deterministic reach report fails if they stop. A second property, 256 cases, applies whole plans to real registry files and install folders through the real removal call and checks the install in use survives. The checked-in regression seed is a case-twin input that fails when the runtime selector fix from #497 is reverted. docs/testing.md gains a "Removing Old Installs" section listing the commands. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The persisted proptest seed is an RNG seed rather than the input, so a change to the generators makes it replay a different case. Pin the case-twin pair the resolver check exists for as an explicit example, asserting first that the policy selects both twins, and say plainly that generated cases only exercise the resolver's exact-match precedence. Also from review: - add a property that the install the configured default names is never selected; dropping only the default hold now fails it at --keep 0; - check that the whole-plan property removes every install the plan named and leaves every other install's folder in place, so neither a removal that does nothing nor one that takes a neighbour with it can pass, and clean its temp root on every path with a drop guard; - fail the hold-reason property with a named assertion instead of a map index panic; - keep each reach-report row's label and floor in one table keyed by a Row enum, with an order check, instead of three positional arrays; - update the docs/testing.md section to match and rewrap it. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
d5a72f1 to
aa23004
Compare
|
Rebased onto main and addressed in aa23004. @juhovainio: there was one rebase conflict, in docs/testing.md, where both sides added a section before "Provider-Assisted Planning"; I kept both. On the approving review's follow-ups:
On the automated review: the |
juhovainio
left a comment
There was a problem hiding this comment.
Test and docs only, and the diff matches the description. No blocking correctness issues found; requesting changes for the seed-file inconsistency below, which contradicts the PR's own account.
- The committed
proptest-regressions/storage/properties.txtcontradicts the module docs, the example-test comment and the mutation table, which all say the #497 case is pinned explicitly rather than left to a seed file. The seed is an RNG replay that changes meaning when a generator changes. Drop it or explain why it stays. - The resolver property always takes the exact-key branch (as its own doc admits), so only the pinned example reaches the case-twin path.
- The hold-reason oracle mirrors the policy's rule order and case-insensitive pointer handling, so it can agree with the policy by construction. Worth a sentence on why that is acceptable, or an independently stated expectation.
- Minor docs nit in
docs/testing.mdon thestorage::propertiesfilter (inline).
CI: all green except E2E tests (Strix Halo, Ubuntu, release, current), which failed at scenario automations-11 (background helper stop), unrelated to a storage-only change; a rerun looks appropriate. I did not build or run the tests, so the mutation table is unverified on my side.
| # | ||
| # It is recommended to check this file in to source control so that | ||
| # everyone who runs the test benefits from these saved cases. | ||
| cc 4cab42fe7419af3e985b4da3f1ad9faff893c57b8dc633d49f6c4b87054ca7dd # shrinks to manifests = [InstalledRuntimeManifest { runtime_key: "A", runtime_id: "therock-release:gfx110X-all", channel: "release", format: "wheel", family: "gfx110X-all", family_source: "test", version: "7.0.0", install_root: "rocm-storage-proptest/A", selected_artifact_url: "https://example.invalid/therock", source_layout_generation: None, index_url: None, tarball_file_name: None, python_launcher: None, python_executable: None, pip_cache_dir: None, rocm_sdk: None, sdk_torch: None, wheel_composition: Some(WheelRuntimeComposition { source_layout_generation: "canonical", package_specs: ["rocm[libraries,devel,device-gfx1100]==7.0.0"], rocm_sdk_target: Some("gfx1100") }), read_only: false, imported_from: None, devel: true, installed_at_unix_ms: 0 }, InstalledRuntimeManifest { runtime_key: "a", runtime_id: "therock-release:gfx110X-all", channel: "release", format: "wheel", family: "gfx110X-all", family_source: "test", version: "7.0.0", install_root: "rocm-storage-proptest/a", selected_artifact_url: "https://example.invalid/therock", source_layout_generation: None, index_url: None, tarball_file_name: None, python_launcher: None, python_executable: None, pip_cache_dir: None, rocm_sdk: None, sdk_torch: None, wheel_composition: Some(WheelRuntimeComposition { source_layout_generation: "canonical", package_specs: ["rocm[libraries,devel,device-gfx1100]==7.0.0"], rocm_sdk_target: Some("gfx1100") }), read_only: false, imported_from: None, devel: true, installed_at_unix_ms: 0 }], inputs = RetentionInputs { active_runtime_key: None, previous_runtime_key: None, default_runtime_id: None, marker_runtime_keys: [] }, keep = 0 |
There was a problem hiding this comment.
The new example test and the module docs say the #497 case is pinned explicitly "rather than left to the seed file", and the PR's mutation table runs with the "seed file removed". This seed is an RNG replay that changes meaning whenever a generator changes. Drop it, or say why it stays.
There was a problem hiding this comment.
Dropped in fb50ca6. You're right that keeping it contradicted the module's own account. It only held the #497 case as an RNG seed, which is redundant now that the literal example pins that input, and after the file move it would also have been an orphan for main's new proptest_regressions_contract check. The PR body's mutation row now says "no seed file", and both a_case_twin_pair_resolves_to_the_install_the_policy_cleared and the generated resolver property fail with the #497 fix reverted.
| /// whose keys differ only in letter case, is pinned as an explicit input in | ||
| /// `a_case_twin_pair_resolves_to_the_install_the_policy_cleared`. | ||
| #[test] | ||
| fn the_deleter_resolves_each_selected_key_to_the_install_the_policy_cleared( |
There was a problem hiding this comment.
Per the doc above, every generated case takes the exact-key branch, so this property cannot reach the case-twin path; only the pinned example does. Consider a generator that emits a case-twin pair deliberately, or fewer cases here.
There was a problem hiding this comment.
Checked by mutation, and this turned out to be a wording problem, fixed in fb50ca6. With the seed file gone, I reverted select_runtime_manifest to try the case-insensitive match first (the #497 defect). That fails this generated property as well as the pinned example: the generators produce keys that differ only in case (generator_reach_report's case-twin rows), and under the reverted precedence one of them resolves to its twin. So the property never reaches the fallback, but #497 was about precedence, and the property does reach that. The doc comment, docs/testing.md and the PR body now say this instead of implying only the example covers it. I kept 2048 cases.
| cargo test -p rocm --bin rocm version_proptests | ||
| ``` | ||
|
|
||
| `storage::properties` runs each property at 2048 cases. All but one check the |
There was a problem hiding this comment.
storage::properties also matches the 20,000-sample generator_reach_report and the pinned example test, which are not 2048-case properties; one clause saying so would avoid surprise.
There was a problem hiding this comment.
Fixed in fb50ca6. The section now says the storage::properties filter also runs generator_reach_report and the pinned example test, which are not properties. It also gives both case counts (2048, and 256 for the whole-plan property) and says PROPTEST_CASES can raise them.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head aa23004f3f82
Outcome: needs work. Test/docs-only change with a sound property-test design, but two CI-contract gaps on the new file plus a currently-failing check need to be resolved before merge.
Blocking
-
New file
apps/rocm/src/storage/properties.rswill fail the repo'scargo xtask check-architecture-docgate (runs in theclippyCI job) —apps/rocm/src/storage/properties.rs,apps/rocm/src/storage.rs:1017
xtask/src/architecture_doc.rs::is_test_helperonly exempts a file whose stem is exactlytest_support/test_env/proptests, or ends with_proptests/_props/_properties. This file's stem is bareproperties— the exact case the check's own unit test (helper_matching_is_by_whole_file_stem) asserts is not a helper (using the identical-stem exampleapps/rocm/src/properties.rs).docs/architecture.mdhas no entry for it, andEXEMPT/EXEMPT_CEILINGare both empty. This gate (added onmainafter this PR's base, and after this PR's last CI run) runs as part ofclippy, so it did not catch this yet — but the PR already needs a rebase onto currentmain(noted elsewhere on this PR), and that rebase will pull the gate in. Fix: rename to match the sibling convention used elsewhere in this codebase (e.g. a flatstorage_properties.rswith#[path = "storage_properties.rs"] mod properties;, mirroringtherock_version_proptests.rs), or add a citation indocs/architecture.md.
Confidence 85 · mechanical -
New proptest case counts are hardcoded, bypassing the project's documented
PROPTEST_CASESoverride contract —apps/rocm/src/storage/properties.rs(cases: 2048),apps/rocm/src/storage.rs(cases: 256, the newapplying_a_whole_plan_never_removes_the_install_in_use)
docs/testing.md's pre-existing "Property tests" section documents that any module setting an explicit case count "reads the variable back itself" throughproptest_cases/case_count_from_env(or the uninstall suite's ownROCM_DELETION_PROP_CASESknob), soPROPTEST_CASES=10000can raise any module's count for a deeper run. Both new proptest blocks use a literalProptestConfig { cases: N, .. }instead of calling that helper, so they're silently exempt from a documented, universal mechanism. Fix: route both throughproptest_cases(2048)/proptest_cases(256)(or a dedicated env knob for the 256-case filesystem one, matching theROCM_DELETION_PROP_CASESpattern).
Confidence 80 · mechanical -
A required-looking CI check is currently failing on this commit —
E2E tests (Strix Halo, Ubuntu, release, current)
The job log showsFAIL: 'automations-helper-stops-when-no-watcher-is-enabled' was expected to pass on this host but FAILED — a regression, exit code 1. This scenario is unrelated to storage/retention, so it reads as a pre-existing flake or host issue rather than something this diff caused — but per CI policy a red check still blocks until it's confirmed green by a rerun.
Confidence 100 (observed directly in the job log) · the fix is to rerun the job, not to change this PR
Decisions for the author
- The one resolver-calling property only ever exercises the exact-match branch today — tradeoff
the_deleter_resolves_each_selected_key_to_the_install_the_policy_clearedcalls the realselect_runtime_manifestrather than re-deriving its rule (reasonable — there's no way to check "the resolver resolves to itself" without either calling it or duplicating its logic). With today's resolver, every freshly-generated case takes the exact-match branch first, so the case-insensitive fallback this property is nominally probing is reached only by the pinned example test, not by the 2048 generated cases. This is disclosed in the property's own doc comment, so it reads as an accepted, understood tradeoff rather than an oversight — flagging so it's explicitly confirmed as such.
Positive signals
- The oracle functions (
expected_group,expected_default_key,held_regardless_of_recency,installed_the_toolchain) are genuinely independently re-derived from the--keephelp text and matching rules rather than thin wrappers around the production helpers — verified line-by-line againstretention_group/resolved_default_runtime_key/unconditional_hold/includes_devel. generator_reach_reportself-checks that itsRowenum and floor table stay in the same order (assert_eq!(*row as usize, index, ...)), closing off the classic silent-misalignment failure mode for this kind of coverage diagnostic.applying_a_whole_plan_never_removes_the_install_in_usechecks both directions (nothing planned is left behind, nothing unplanned disappears) against real registry files and real install folders, not just the easier "survivor" direction.docs/testing.md's new section was cross-checked command-by-command and test-name-by-test-name against the actual code and is accurate.
What this covered
Full review of the whole diff (88c344d1…aa23004f): the new apps/rocm/src/storage/properties.rs, the diff to apps/rocm/src/storage.rs, docs/testing.md, and the new proptest-regressions seed file, plus the production code under test (runtime_registry.rs, runtimes_cmd.rs, therock.rs, therock_version_proptests.rs), repo AGENTS.md, and relevant git history (including the PR #497 fix this suite guards). Four independent fan-out passes (core correctness, docs/fixtures, agent-instructions + code comments, history + prior changes) plus a direct read of xtask/src/architecture_doc.rs, docs/testing.md's property-test conventions section, and the PR's live CI checks/logs. No ticket named; description's scope matches the diff.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
juhovainio
left a comment
There was a problem hiding this comment.
$(cat /tmp/review_body.md)
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod properties; |
There was a problem hiding this comment.
Blocking: this new module's file will fail the repo's architecture-doc gate. apps/rocm/src/storage/properties.rs's stem is the bare properties, which xtask/src/architecture_doc.rs's is_test_helper() does not exempt — its own unit test (helper_matching_is_by_whole_file_stem) asserts exactly this stem (via the identical example apps/rocm/src/properties.rs) must not count as a helper. docs/architecture.md has no entry for this file on this PR's head or on current main. The checking file isn't present at this PR's head/merge-base yet (only on main's current tip), so the branch's last CI run never exercised this gate — but it already runs in the clippy job on main and will apply once this branch rebases, which it needs anyway (mergeable_state is already blocked).
Fix: rename to something like storage_properties.rs with #[path = "storage_properties.rs"] mod properties;, mirroring crates/rocm-core/src/process.rs's #[path = "process_tree_properties.rs"], or add a docs/architecture.md citation for the file as it stands.
Confidence 90 · mechanical.
There was a problem hiding this comment.
Fixed in fb50ca6. The file is now apps/rocm/src/storage_properties.rs, declared from storage.rs as #[path = "storage_properties.rs"] mod properties; (the process_tree_properties.rs pattern), so test paths and the documented storage::properties filter stay the same. I also added a docs/architecture.md row for it, like its sibling property files have. On the branch merged with current main, cargo xtask check-architecture-doc passes.
| saw[Row::GroupFamilyTwoWays as usize] |= left.family != right.family; | ||
| } | ||
| } | ||
| if members.len() > 1 { |
There was a problem hiding this comment.
Non-blocking: these single-manifest reach rows undercount singleton groups. Rows that need a pair (tied install times, case-twin keys, channel/format/family spelled two ways, computed a few lines above this if) are correctly gated behind members.len() > 1. The devel-disagrees, spaced-extra, near-miss-extra and other-package's-spec-first rows below this line are single-manifest .any() checks nested inside the same guard, so a retention group of exactly one install exhibiting any of those conditions is silently excluded from the reach count. No proptest! property is affected — those run over the whole generated group regardless of size — but the measured floor for these four rows under-counts true generator reach.
Fix: move those four .any() checks outside the members.len() > 1 guard, or give them their own ungated block.
Confidence 80 · mechanical.
There was a problem hiding this comment.
Fixed in fb50ca6. The four single-install toolchain rows (devel flag disagreeing with the specs, spaced extra, near-miss extra, another package's spec first) are now counted for every retention group, including a group of one. A lone install whose toolchain is misread can still move into a sibling group, so the old members.len() > 1 gate undercounted, as you said. I re-measured the floors at four fifths of the new counts (11638 → 9_310, 3427 → 2_740, 3424 → 2_740, 6104 → 4_880). The pairwise rows stay behind the gate.
| } | ||
|
|
||
| proptest! { | ||
| #![proptest_config(ProptestConfig { cases: 2048, ..ProptestConfig::default() })] |
There was a problem hiding this comment.
Non-blocking: this hardcoded case count bypasses the documented PROPTEST_CASES override helper. docs/testing.md's "Property tests" section documents that a module setting an explicit count is expected to read PROPTEST_CASES back itself through the proptest_cases helper. This is a literal cases: 2048 instead. Flagging as non-blocking rather than blocking: the sibling module this one cross-references, apps/rocm/src/therock_version_proptests.rs, already hardcodes cases: 4096/cases: 2048 the same way at six call sites without the helper either, so this follows existing practice in the same area rather than introducing a fresh gap.
There was a problem hiding this comment.
Fixed in fb50ca6: this block now uses ProptestConfig::with_cases(proptest_cases(2048)), so PROPTEST_CASES can raise it (never lower it). docs/testing.md says so for this suite.
| /// day to day is that precedence. The case it exists for, two installs | ||
| /// whose keys differ only in letter case, is pinned as an explicit input in | ||
| /// `a_case_twin_pair_resolves_to_the_install_the_policy_cleared`. | ||
| #[test] |
There was a problem hiding this comment.
Decision for the author (tradeoff): this property only ever exercises the exact-match branch today. Calling the real select_runtime_manifest rather than re-deriving its rule is the right call — there's no way to check "the resolver resolves to itself" without either calling it or duplicating its logic. With today's resolver, every freshly generated case takes the exact-match branch first, so the case-insensitive fallback this property nominally probes is reached only by the pinned example test, not by the 2048 generated cases. The doc comment above already discloses this, so it reads as an accepted tradeoff rather than an oversight — flagging so it's explicitly confirmed as such.
There was a problem hiding this comment.
Confirmed as the intended tradeoff, with a correction in fb50ca6. The property calls the real resolver and so never reaches its case-insensitive fallback. But a resolver that tries that fallback first, the #497 defect, does fail this property on the generated case twins; I checked by reverting it with no seed file present. The doc comment now says it guards the exact-match-first precedence, and that the literal example pins the same input independently of the generators.
|
|
||
| /// The input that #497's selector fix is about, pinned here rather than left | ||
| /// to the proptest seed file: that file stores an RNG seed, so a change to the | ||
| /// generators makes it replay a different case. Two installs whose keys differ |
There was a problem hiding this comment.
Decision for the author (tradeoff): the committed proptest-regressions seed is kept even though this input is now pinned explicitly. This comment already explains why: a seed replays an RNG-generated case rather than a fixed input, so it would replay something else once the generators' shape changes, which is why the #497 case moved to a literal example test instead. The seed file still exists alongside that pinned example. Since the reasoning is already stated here, this reads as a deliberate choice (kept for its own historical replay value) rather than an unexplained inconsistency — worth a one-line confirmation that keeping it serves a purpose beyond habit.
|
Correction: the review above (pull request review #5936587331) was submitted with a tooling error — its body rendered as the literal text Review — needs workFull review of the whole diff ( Blocking
Non-blocking
Decisions for the author
Positive signals
Deployment notesNone — test and documentation only, no dependency, migration or behavior change. What this coveredRead completely: the new Four earlier reviews exist at or near this head, surfaced by a required duplicate-review check before this one started, so this review did not begin fully blind of their content; every point reconciled above (the architecture-doc gate, the reach-report gating, the case-count convention, the two tradeoffs) was independently re-derived and verified against the source, CI logs, and git history rather than taken on their word, and several of their already-raised points (a temp-directory leak on early return, a possible panic on a missing map entry, the reach-report's row/label/floor ordering, and an assertion that only checked one direction of removal) read as already fixed in this exact commit. Not reconciled against any PR discussion thread beyond the review bodies themselves. 🤖 Automated by an AI agent |
juhovainio
left a comment
There was a problem hiding this comment.
Review — needs work
Full review of the whole change (test/docs-only; no production code touched). No ticket named; the description references PR #497 as prior context for the bug this suite guards, not as a ticket this PR closes. Description's scope ("test and docs only; no production code changes") matches the diff.
Blocking
- A CI check is currently red on this commit —
E2E tests (Strix Halo, Ubuntu, release, current)
The job log for this exact head (aa23004f) showsFAIL: 'automations-helper-stops-when-no-watcher-is-enabled' was expected to pass on this host but FAILED — a regression, exit code 1. The failing scenario is an automations/watcher-stop check, unrelated to storage/retention, so it reads as a pre-existing flake or host issue rather than something this diff caused — but it is still red today.
Confidence 100 (read directly from the job log) · Fix: rerun the job to confirm green before merge.
Non-blocking
- Both new proptest suites hardcode a literal case count instead of the crate's
proptest_cases()env-override helper —apps/rocm/src/storage.rs:1960,apps/rocm/src/storage/properties.rs:384
docs/testing.md's pre-existing "Property tests" section documents that a module setting an explicit count "reads the variable back itself" through theproptest_caseshelper (apps/rocm/src/test_support.rs:61), used bygpu_selection_proptests.rs,providers/local_http/proptests.rs,providers/wire/sse_emitter_proptests.rsandcrates/rocm-core/src/examine_driver_proptests.rs. Both of this PR's new blocks use a bareProptestConfig { cases: N, .. }instead, soPROPTEST_CASES=10000— the exact command this PR's own new docs section recommends for other properties — silently has no effect on either new suite. This isn't a deviation unique to this PR, though: the pre-existing, untouchedapps/rocm/src/therock_version_proptests.rs(six blocks,cases: 2048/4096) already uses the identical bare-literal pattern, so the docs' "all of them" framing was already inaccurate before this change, and this PR extends an existing (if undocumented) precedent rather than inventing a new one.
Confidence 80 · mechanical · Fix: route both throughproptest_cases(2048)/proptest_cases(256)for consistency with the helper-based suites, or note the literal-config pattern as a second accepted style indocs/testing.md.
Decisions for the author
- The architecture-doc gate doesn't exist in this PR's history today, but will apply on the next rebase — tradeoff
xtask/src/architecture_doc.rsis absent at both this PR's base and its tip (confirmed directly); it was added tomainafter this PR diverged.git merge-treebetween currentorigin/mainand this PR's tip resolves with no conflicts, so a rebase today would bring the checker in alongside the newapps/rocm/src/storage/properties.rs. That checker'sis_test_helpermatches by whole file stem, and its own unit test explicitly treats a bareproperties.rsstem as not exempt — this file's stem is exactlyproperties, so it will need adocs/architecture.mdcitation or a rename (e.g.storage_properties.rs) to pass post-rebase. Flagging so it's handled at rebase time rather than discovered by a redclippyjob afterward. - The one property that calls the real resolver only ever exercises its exact-match branch on freshly generated cases — tradeoff
the_deleter_resolves_each_selected_key_to_the_install_the_policy_clearedcalls the realselect_runtime_manifest(verified against the actual#497fix commit: it matchesruntime_key == selectorbefore any case-insensitive fallback), andmanifests()only dedupes exact keys, so the case-insensitive branch this property is nominally about is reached only by the hand-pinneda_case_twin_pair_resolves_to_the_install_the_policy_cleared, not by the 2048 generated cases. This is disclosed accurately in both the property's own doc comment and the PR description, so it reads as an understood, deliberate tradeoff rather than an oversight.
Positive signals
- The oracle functions in
storage/properties.rs(expected_default_key,held_regardless_of_recency,installed_the_toolchain,expected_group) are genuine independent reimplementations ofresolved_default_runtime_key/unconditional_hold/includes_devel/retention_group, traced line-by-line and found faithful, including the case-insensitive-pointer and devel-flag/package-spec-disagreement rules. generator_reach_reportself-checks its ownRowenum against its floor table's array order (assert_eq!(*row as usize, index, ...)), closing off the classic silent-misalignment failure mode for this kind of diagnostic.RemoveOnDrop(storage.rs) cleans up its scratch tree on every exit path, including an early?— a real improvement over the pre-existing hand-written tests in the same file, which only remove their root as the last line and leak it if an earlier assertion aborts.applying_a_whole_plan_never_removes_the_install_in_usechecks both directions (nothing planned is left behind, nothing unplanned disappears) against real registry files and real folders, and its doc comment accurately discloses what it does not re-check (the production loop's last-second hold re-check, left to the existingprune_removes_the_install_it_named_and_not_a_case_twin_of_it).docs/testing.md's new section gives exact, runnable commands and accurately states the case counts actually present in the code.
Deployment notes
None — test and documentation only, no production code changed.
What this covered
Full review of the whole diff (88c344d1…aa23004f): fanned out two independent passes — one reading the new apps/rocm/src/storage/properties.rs (979 lines) completely against its production oracle (runtime_registry.rs, therock.rs) and the regression seed file; one reading the diff and the full apps/rocm/src/storage.rs (2026 lines) against runtime_uninstall.rs and select_runtime_manifest's contract in runtime_registry.rs — plus a third pass covering docs/testing.md's diff and its pre-existing Property-tests section, the root AGENTS.md, git history of the #497 fix this suite guards, this commit's live CI check list and the one failing job's log, a direct read of xtask/src/architecture_doc.rs and its own unit test, and a conflict-free git merge-tree check between this PR's tip and current origin/main.
Did not run cargo test/cargo xtask coverage (no Rust toolchain in this environment); the PR's own "34/34 passing" claim and mutation-evidence table are read from the PR description and from CI's green build-and-test/Test (affected crates)/clippy results, taken at face value rather than independently executed.
Not reconciled against any prior review or discussion on this PR; this is a blind pass, cross-checked against the code, git history (including the actual #497 fix commit) and a live git merge-tree, not against anyone's earlier conclusions.
🤖 Automated by an AI agent
| // Each case writes and deletes a tree, so this matches the other | ||
| // filesystem-backed property suites rather than the pure-policy | ||
| // budget. | ||
| cases: 256, |
There was a problem hiding this comment.
Non-blocking: this hardcodes cases: 256 instead of going through the crate's proptest_cases() helper (apps/rocm/src/test_support.rs:61), which every other env-scalable proptest suite in the crate uses (gpu_selection_proptests.rs, providers/local_http/proptests.rs, providers/wire/sse_emitter_proptests.rs, crates/rocm-core/src/examine_driver_proptests.rs). docs/testing.md's "Property tests" section documents this as universal, so PROPTEST_CASES=10000 silently has no effect on this suite.
That said, this isn't a novel deviation: the pre-existing, untouched apps/rocm/src/therock_version_proptests.rs already uses the same bare ProptestConfig { cases: N, .. } pattern (six blocks), so the docs' "all of them" framing was already inaccurate before this PR. Worth routing through proptest_cases(256) for consistency, or acknowledging the literal-config pattern as a second accepted style in the docs.
There was a problem hiding this comment.
Fixed in fb50ca6: ProptestConfig::with_cases(crate::test_support::proptest_cases(256)). I left therock_version_proptests.rs's bare literals for a separate change, since that file is outside this PR.
| } | ||
|
|
||
| proptest! { | ||
| #![proptest_config(ProptestConfig { cases: 2048, ..ProptestConfig::default() })] |
There was a problem hiding this comment.
Same pattern as the new storage.rs property (see that inline comment): cases: 2048 is a bare literal rather than proptest_cases(2048), so PROPTEST_CASES doesn't scale this suite either. Non-blocking given the pre-existing therock_version_proptests.rs precedent for the same bare-literal style, but worth aligning with the helper-based suites for consistency.
Separately (tradeoff, not a defect): xtask/src/architecture_doc.rs doesn't exist anywhere in this PR's history today (confirmed absent at both base and tip), so this file isn't currently gated — but it was added to main after this PR's base, and git merge-tree shows this branch merges cleanly with current main, so the next rebase will pull that checker in. Its is_test_helper matches by whole file stem, and its own unit test explicitly treats a bare properties.rs stem as not exempt — this file's stem is exactly properties, so it will need either a docs/architecture.md citation or a rename (e.g. storage_properties.rs) to pass post-rebase.
There was a problem hiding this comment.
Both points are fixed in fb50ca6: the case count goes through proptest_cases(2048), and the file is renamed to storage_properties.rs with an architecture-doc row. cargo xtask check-architecture-doc passes on the branch merged with current main. See the thread on storage.rs for details.
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…ew points Main gained an architecture-doc check that exempts property-test files only by a `_properties`-style stem, so the bare `storage/properties.rs` would fail it. Move it to `storage_properties.rs`, declared through `#[path]` as the same `properties` module (as `process_tree_properties.rs` is), so test paths and the documented `storage::properties` filter are unchanged, and give it an architecture row like its sibling property files. Drop the saved proptest seed instead of carrying it to the new name. It held the #497 case-twin failure as an RNG seed, which replays a different input once the generators change; that input is pinned as the literal example `a_case_twin_pair_resolves_to_the_install_the_policy_cleared`, so the seed was redundant, and left in place it would also be orphaned by the move. Read both suites' case counts through `proptest_cases`, so `PROPTEST_CASES` can raise them as it does for the other suites. Count the four single-install toolchain rows of `generator_reach_report` in groups of one too: a lone install whose toolchain is misread can still move into a sibling group. Floors re-measured at four fifths of the new counts. Correct what the resolver property is said to guard: generated case twins do fail it when the resolver tries a case-insensitive match first (checked by reverting that precedence), so it is not only the pinned example that covers #497. State why the hold-reason oracle follows the policy's own order, and say which tests the `storage::properties` filter runs besides properties. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
Why
rocm storage remove-old-installsdeletes multi-gigabyte install folders from a set the user never listed one by one, so a retention bug destroys data the user did not choose to lose. The example tests instorage::testspin specific scenarios, but the dangerous inputs are combinations: pointers spelled in a different case, two installs whose keys differ only in case, a whole family sharing one runtime id, tied install times, a pointer naming nothing,--keep 0. Property tests over those combinations are what found the case-twin runtime-key defect fixed in #497; this change lands the suite itself so it keeps guarding the policy.What changes
Property tests for the pure retention policy (
select_runtimes_to_remove), inapps/rocm/src/storage_properties.rs, 2048 cases each (PROPTEST_CASEScan raise the count throughproptest_cases):--keep 0;--keepmost recent installs that nothing else holds (ties broken by key) and removes the rest; raising--keepnever removes more;HoldReasondeclares them;select_runtime_manifest, to that same install rather than to a case twin of it. The resolver tries an exact key match first, so generated cases never reach its case-insensitive fallback; what this guards is that precedence, which is what fix(runtimes): resolve a runtime selector to the key that matches exactly #497 fixed. Generated keys that differ only in case fail it when the resolver tries the case-insensitive match first, and the same input is also pinned as the example testa_case_twin_pair_resolves_to_the_install_the_policy_cleared.The expected groups and holds are computed in the test from the
--keephelp and the matching rules the policy implements, not by calling the policy's helpers, so a defect in one of those helpers cannot hide itself. Generators draw from small, weighted alphabets so the rare inputs above actually occur, plus toolchain records whosedevelflag disagrees with their recorded package specs;generator_reach_reportis a deterministic measurement that fails if the generators stop reaching them; each row's label and floor sit together in one table keyed by aRowenum, and the floors need re-measuring after any generator change.A second property (256 cases, also raisable through
PROPTEST_CASES) applies whole generated plans to real registry files and install folders through the real removal call and checks both directions: the install in use and every other install the plan left out keep their folders, and every install the plan named is gone. It calls the removal per key rather than throughstorage()'s loop, so that loop's last-second hold re-check is covered by the hand-writtenprune_removes_the_install_it_named_and_not_a_case_twin_of_itinstead.The case-twin input from #497 is pinned as the example test
a_case_twin_pair_resolves_to_the_install_the_policy_cleared, which fails when that fix is reverted. No proptest seed file is committed: a seed stores an RNG state and replays a different input once the generators change, so the literal example replaces it.docs/testing.mdgains a "Removing Old Installs" section with the commands.docs/architecture.mdgains a row for the property-test file, which sits besidestorage.rsunder a_propertiesname so the architecture-doc check onmaintreats it as a test helper.Verification
main:cargo fmt --all --checkandcargo clippy --workspace --all-targets -- -D warningsare clean;cargo xtask check-architecture-docandcargo xtask check-crate-edgespass;cargo test -p rocm --bin rocm storage::passes 40/40.cargo test --workspace --all-targets --no-fail-fast: 7566 passed. The 5 failures are local-environment only and pass where the environment allows: twobuild.rsgit-fixture tests that the local global git signing config breaks (pass withGIT_CONFIG_GLOBAL=/dev/null), and three that depend on whereTMPDIRpoints or on a podman-backeddockershim. None touches storage.cargo xtask smokepasses up torocmd sandbox examine, which bubblewrap cannot run inside the local sandbox.select_runtime_manifestresolves a case-insensitive match first (the defect #497 fixed); no seed filea_case_twin_pair_resolves_to_the_install_the_policy_cleared,the_deleter_resolves_each_selected_key_to_the_install_the_policy_cleareddevelflageach_group_keeps_its_most_recent_installs_and_removes_the_resteach_group_keeps_…every_hold_reason_is_the_first_that_applies,each_group_keeps_…every_hold_reason_is_the_first_that_appliesevery_hold_reason_is_the_first_that_appliesthe_install_in_use_is_never_selected_for_removal,applying_a_whole_plan_never_removes_the_install_in_usethe_install_the_default_names_is_never_selected_for_removal,each_group_keeps_…,every_hold_reason_is_…applying_a_whole_plan_never_removes_the_install_in_use(planned installs left behind)applying_a_whole_plan_never_removes_the_install_in_use(unplanned installs gone)generator_reach_report(order check)develextra matched bycontains, withouttrim, or under any package's[each_group_keeps_…,every_hold_reason_is_…Not covered:
RetentionInputs::from_config(how config and the marker file become pointers) — the properties buildRetentionInputsdirectly. Extras spelledDeveland requirements on otherrocm-*packages are not generated.Scope
Test and docs only; no production code changes. No e2e scenario is added because no user-observable behaviour changes. The retention properties in
therock::version_proptests(order independence, recency over version) are left where they are; the new module cross-references them.tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. — N/A, test-only.docs/architecture.md. — N/A.