Skip to content

test(storage): property-test the prune retention invariants - #815

Open
rominf wants to merge 4 commits into
mainfrom
test/proptest-storage-retention
Open

rominf wants to merge 4 commits into
mainfrom
test/proptest-storage-retention

Conversation

@rominf

@rominf rominf commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Why

rocm storage remove-old-installs deletes multi-gigabyte install folders from a set the user never listed one by one, so a retention bug destroys data the user did not choose to lose. The example tests in storage::tests pin specific scenarios, but the dangerous inputs are combinations: pointers spelled in a different case, two installs whose keys differ only in case, a whole family sharing one runtime id, tied install times, a pointer naming nothing, --keep 0. Property tests over those combinations are what found the case-twin runtime-key defect fixed in #497; this change lands the suite itself so it keeps guarding the policy.

What changes

Property tests for the pure retention policy (select_runtimes_to_remove), in apps/rocm/src/storage_properties.rs, 2048 cases each (PROPTEST_CASES can raise the count through proptest_cases):

  • the install in use, the rollback target, the install the configured default names (when it names exactly one), anything the active-runtime marker names and anything ROCm CLI does not own are never selected, including at --keep 0;
  • each retention group keeps exactly its --keep most recent installs that nothing else holds (ties broken by key) and removes the rest; raising --keep never removes more;
  • every install is kept or removed exactly once, and each kept install is reported with the first hold reason that applies, in the order HoldReason declares them;
  • every selected key resolves, through the real select_runtime_manifest, to that same install rather than to a case twin of it. The resolver tries an exact key match first, so generated cases never reach its case-insensitive fallback; what this guards is that precedence, which is what fix(runtimes): resolve a runtime selector to the key that matches exactly #497 fixed. Generated keys that differ only in case fail it when the resolver tries the case-insensitive match first, and the same input is also pinned as the example test a_case_twin_pair_resolves_to_the_install_the_policy_cleared.

The expected groups and holds are computed in the test from the --keep help and the matching rules the policy implements, not by calling the policy's helpers, so a defect in one of those helpers cannot hide itself. Generators draw from small, weighted alphabets so the rare inputs above actually occur, plus toolchain records whose devel flag disagrees with their recorded package specs; generator_reach_report is a deterministic measurement that fails if the generators stop reaching them; each row's label and floor sit together in one table keyed by a Row enum, and the floors need re-measuring after any generator change.

A second property (256 cases, also raisable through PROPTEST_CASES) applies whole generated plans to real registry files and install folders through the real removal call and checks both directions: the install in use and every other install the plan left out keep their folders, and every install the plan named is gone. It calls the removal per key rather than through storage()'s loop, so that loop's last-second hold re-check is covered by the hand-written prune_removes_the_install_it_named_and_not_a_case_twin_of_it instead.

The case-twin input from #497 is pinned as the example test a_case_twin_pair_resolves_to_the_install_the_policy_cleared, which fails when that fix is reverted. No proptest seed file is committed: a seed stores an RNG state and replays a different input once the generators change, so the literal example replaces it. docs/testing.md gains a "Removing Old Installs" section with the commands. docs/architecture.md gains a row for the property-test file, which sits beside storage.rs under a _properties name so the architecture-doc check on main treats it as a test helper.

Verification

  • On the branch merged with current main: cargo fmt --all --check and cargo clippy --workspace --all-targets -- -D warnings are clean; cargo xtask check-architecture-doc and cargo xtask check-crate-edges pass; cargo test -p rocm --bin rocm storage:: passes 40/40.
  • cargo test --workspace --all-targets --no-fail-fast: 7566 passed. The 5 failures are local-environment only and pass where the environment allows: two build.rs git-fixture tests that the local global git signing config breaks (pass with GIT_CONFIG_GLOBAL=/dev/null), and three that depend on where TMPDIR points or on a podman-backed docker shim. None touches storage. cargo xtask smoke passes up to rocmd sandbox examine, which bubblewrap cannot run inside the local sandbox.
  • Mutation evidence — each deliberate defect below makes at least one new test fail:
Deliberate defect in production code Caught by
select_runtime_manifest resolves a case-insensitive match first (the defect #497 fixed); no seed file a_case_twin_pair_resolves_to_the_install_the_policy_cleared, the_deleter_resolves_each_selected_key_to_the_install_the_policy_cleared
retention group ignores the toolchain axis, or reads only the devel flag each_group_keeps_its_most_recent_installs_and_removes_the_rest
keep everything / keep one too few / tie-break reversed each_group_keeps_…
default id holds an install when it names several every_hold_reason_is_the_first_that_applies, each_group_keeps_…
hold reasons checked in a different order (3 swaps) every_hold_reason_is_the_first_that_applies
active pointer compared case-sensitively; read-only install not held the matching never-selected property; the case-sensitive pointer also fails every_hold_reason_is_the_first_that_applies
active and default holds dropped the_install_in_use_is_never_selected_for_removal, applying_a_whole_plan_never_removes_the_install_in_use
default hold dropped alone the_install_the_default_names_is_never_selected_for_removal, each_group_keeps_…, every_hold_reason_is_…
removal skips deleting the folder applying_a_whole_plan_never_removes_the_install_in_use (planned installs left behind)
removal deletes the shared parent folder (in-use check disabled) applying_a_whole_plan_never_removes_the_install_in_use (unplanned installs gone)
two reach-report rows swapped generator_reach_report (order check)
devel extra matched by contains, without trim, or under any package's [ each_group_keeps_…, every_hold_reason_is_…

Not covered: RetentionInputs::from_config (how config and the marker file become pointers) — the properties build RetentionInputs directly. Extras spelled Devel and requirements on other rocm-* packages are not generated.

Scope

Test and docs only; no production code changes. No e2e scenario is added because no user-observable behaviour changes. The retention properties in therock::version_proptests (order independence, recency over version) are left where they are; the new module cross-references them.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. — N/A, test-only.
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. — N/A.
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state. — N/A, no user-facing messages changed.

@rominf
rominf marked this pull request as ready for review November 3, 2026 15:18
@rominf
rominf requested a review from a team as a code owner November 3, 2026 15:18
@rominf
rominf requested a review from juhovainio November 3, 2026 15:18

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · d5a72f1

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (test-only: no production code touched).
No ticket named. Description references PR #497 as prior context only, not as a ticket this PR closes.

Blocking

None

Non-blocking

  • generator_reach_report's rows 12–15 undercount singleton retention groups — apps/rocm/src/storage/properties.rs:727-747
    Rows 7–11 (tied install times, case-twin keys, channel/format/family spelled two ways) are inherently pairwise and correctly gated behind members.len() > 1. Rows 12–15 (devel flag disagreeing with specs, spaced extra, near-miss extra, another package's spec first) are single-manifest .any() checks copied under the same len() > 1 guard, so a group of exactly one install exhibiting any of those conditions is silently excluded from the reach count. This doesn't make any proptest! property vacuous — those run over the full generated group regardless of size — but it means the floor for rows 12–15 under-measures true generator reach, and a regression that only cost the generator its singleton-group occurrences of these conditions would not trip generator_reach_report.
    Confidence 85 · mechanical · Fix: move the four .any() checks for rows 12–15 outside the members.len() > 1 block (or add a second, ungated block for them).

Decisions for the author

  • the_deleter_resolves_each_selected_key_to_the_install_the_policy_cleared is guaranteed to pass on every freshly-generated case under today's code — non-blocking-improvement
    select_runtime_manifest (apps/rocm/src/runtimes_cmd.rs:1656-1670) matches on runtime_key == selector first and returns immediately; the key this property passes in is always an exact clone of an existing manifest's runtime_key (produced by select_runtimes_to_remove), so the exact-match branch fires on every case the 2048-sample run generates — the case-insensitive fallback this property is nominally probing is never reached by fresh generation. The property's own doc comment is honest about why it still earns its place ("the real resolver is called here… so this keeps holding whatever that rule becomes") and its practical protection is real: it would catch a regression that removed or weakened the exact-match-first precedence, and the checked-in regression seed (apps/rocm/proptest-regressions/storage/properties.txt) replays the historical #497 case-twin failure on every run regardless of fresh generation. Worth confirming this is the understood tradeoff (day-to-day protection comes from the seed plus select_runtime_manifest's own direct tests, not from this property's random cases) rather than something a future reader should assume is a live, generically-exercised cross-check.
  • applying_a_whole_plan_never_removes_the_install_in_use only asserts the negative direction — tradeoff
    apps/rocm/src/storage.rs:1842-1877 builds a plan, executes every uninstall_runtime call, and then only checks that the active install's folder survived — it never asserts that the entries in plan.remove were actually deleted, so a silent no-op in the removal loop (nothing ever gets removed) would still pass. This is a reasonable, deliberate scope choice: removal-actually-happened coverage already exists in the pre-existing hand-written prune_removes_the_install_it_named_and_not_a_case_twin_of_it (apps/rocm/src/storage.rs:1742), which asserts both that the named install's folder and registry entry are gone and that the surviving ones remain. Flagging so the asymmetry is explicit rather than assumed.

Positive signals

  • Re-deriving expected groups/holds independently of the production helpers (expected_default_key, held_regardless_of_recency, installed_the_toolchain, expected_group in properties.rs) rather than calling resolved_default_runtime_key/unconditional_hold/includes_devel/retention_group directly is exactly the right design for a policy this safety-sensitive — traced line-by-line against production and found faithful in every case, including the subtler case-insensitive-everywhere and devel-flag/package-spec disagreement rules.
  • generator_reach_report as a deterministic, floor-checked measurement of generator coverage (not just the properties themselves) is an uncommon and valuable practice that catches the generators silently drifting away from the edge cases the properties depend on.
  • The weighted, tiny alphabets (KEYS, CHANNELS, FORMATS, FAMILIES, the Toolchain enum) are a well-reasoned, explicitly-commented answer to the standard "uniform random strings never collide" failure mode for property tests over anything key-based.

Deployment notes

None — test and documentation only, no dependency, migration, or behavior change.

What this covered

Read completely: apps/rocm/src/storage/properties.rs (new file, 782 lines), the whole of apps/rocm/src/storage.rs (1934 lines, including the new mod properties; declaration and the new AppliedPlan/apply_generated_plan/applying_a_whole_plan_never_removes_the_install_in_use test code), docs/testing.md's new section, and apps/rocm/proptest-regressions/storage/properties.txt. Targeted reads in apps/rocm/src/therock.rs (InstalledRuntimeManifest, WheelRuntimeComposition, includes_devel, wheel_composition_includes_devel, load_runtime_manifests) and apps/rocm/src/runtimes_cmd.rs (select_runtime_manifest, uninstall_runtime), plus the full root AGENTS.md, git history/blame around the touched regions, and PR #497 (the fix this suite guards) for context. Reviewed against commits 9301dfc2…d5a72f1d.

Fanned out two independent passes (storage/properties correctness; docs+history+agent-instruction adherence+impact analysis), consolidated their findings against the source myself, and ran the design-review pass directly (reading select_runtime_manifest and the pre-existing hand-written regression test myself to settle the two items above) rather than spawning further sub-workers for it, since the two code-reading passes had already surfaced the material to decide both questions.

Did not run: cargo test/cargo xtask coverage — no Rust toolchain available in this environment, so the PR's own mutation-evidence table and "31/31 passing" claim are read from CI and the PR description, not independently executed; CI's build-and-test, Test (affected crates), and clippy checks are all green for this exact commit, which is treated at face value. All CI checks for d5a72f1d are SUCCESS or SKIPPED except two self-hosted E2E lanes (E2E tests (Strix Halo, Ubuntu, release/nightly, current)), which show CANCELLED; the self-hosted consolidated report finished SUCCESS afterward and every other self-hosted hardware lane (MI300X, Strix Halo Windows/WSL2, W7900) succeeded, and this change touches no e2e-relevant code, so this reads as unrelated infra rather than a result of this PR — noted rather than assumed. History and prior-changes passes ran (PR #497 and its tests inspected directly). Not reconciled against any PR discussion/comments — none were read, by design.

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks Romain. No blockers: the oracle recomputes groups/holds without the production helpers, the reach report is sound, and the docs counts match.

Rebase note: one conflict with main, in docs/testing.md (both sides add a section before "Provider-Assisted Planning"); keep both. No code conflict; the re-exports the tests use are still there.

Non-blocking, worth a follow-up

  • applying_a_whole_plan_never_removes_the_install_in_use only asserts the active install survives; it would also pass if plan.remove were empty, and it skips the production loop's pre-delete hold re-check. Asserting that planned installs are actually gone (or extracting the production loop) would tighten it.
  • No direct "configured default is never selected" property: at --keep 0, dropping only the default hold survives (the PR's mutation row drops active and default together).
  • The checked-in seed's "replays the case-twin" wording can't be shown from the shrunk value; the hand-written prune_removes_the_install_it_named_and_not_a_case_twin_of_it covers it anyway, so softening the wording is enough.
  • Reach row 19 (floor 130, expected ~160) has a thin margin if the proptest RNG stream ever changes.

🤖 by agenthub on AMD AgentHub

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough property suite; the oracle being derived from the --keep help text rather than the policy's own helpers is a good call, and the reach report is a nice guard against vacuous generators. One blocker and a few smaller points.

Needs fixing before merge

  • The branch currently conflicts with main. After rebasing, please re-run the storage:: tests, since storage.rs / runtimes_cmd.rs may have moved underneath this.

Worth addressing (inline comments below)

  • The persisted proptest seed is an RNG seed, not the input, so the claim that it fails when the #497 fix is reverted isn't actually shown (the mutation check was done with the seed file removed).
  • apply_generated_plan leaks its temp dir on early ? returns.
  • The hold-reason property can panic on a map index instead of giving a readable failure.
  • The saw[..] / labels / FLOOR alignment in the reach report is by position only.

For awareness

  • The 20k-sample reach report has hard-coded floors, so they need re-measuring whenever a generator changes. Deterministic, so not flaky, but a maintenance cost.
  • One line in docs/testing.md is over-long (cosmetic).
  • The two Strix Halo Ubuntu E2E jobs (nightly, release) hit the 45m job limit; that looks like a runner timeout for a test-only PR, but they should be rerun after the rebase to confirm.

#
# It is recommended to check this file in to source control so that
# everyone who runs the test benefits from these saved cases.
cc 4cab42fe7419af3e985b4da3f1ad9faff893c57b8dc633d49f6c4b87054ca7dd # shrinks to manifests = [InstalledRuntimeManifest { runtime_key: "A", runtime_id: "therock-release:gfx110X-all", channel: "release", format: "wheel", family: "gfx110X-all", family_source: "test", version: "7.0.0", install_root: "rocm-storage-proptest/A", selected_artifact_url: "https://example.invalid/therock", source_layout_generation: None, index_url: None, tarball_file_name: None, python_launcher: None, python_executable: None, pip_cache_dir: None, rocm_sdk: None, sdk_torch: None, wheel_composition: Some(WheelRuntimeComposition { source_layout_generation: "canonical", package_specs: ["rocm[libraries,devel,device-gfx1100]==7.0.0"], rocm_sdk_target: Some("gfx1100") }), read_only: false, imported_from: None, devel: true, installed_at_unix_ms: 0 }, InstalledRuntimeManifest { runtime_key: "a", runtime_id: "therock-release:gfx110X-all", channel: "release", format: "wheel", family: "gfx110X-all", family_source: "test", version: "7.0.0", install_root: "rocm-storage-proptest/a", selected_artifact_url: "https://example.invalid/therock", source_layout_generation: None, index_url: None, tarball_file_name: None, python_launcher: None, python_executable: None, pip_cache_dir: None, rocm_sdk: None, sdk_torch: None, wheel_composition: Some(WheelRuntimeComposition { source_layout_generation: "canonical", package_specs: ["rocm[libraries,devel,device-gfx1100]==7.0.0"], rocm_sdk_target: Some("gfx1100") }), read_only: false, imported_from: None, devel: true, installed_at_unix_ms: 0 }], inputs = RetentionInputs { active_runtime_key: None, previous_runtime_key: None, default_runtime_id: None, marker_runtime_keys: [] }, keep = 0

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line stores an RNG seed, not the input; any change to the generators makes it replay a different case. The PR notes the #497 mutation was checked with this file removed, so please either pin the case-twin input as an explicit example test or soften the claim that this seed fails when the fix is reverted.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: the file holds an RNG seed, not the input, so it stops replaying the case-twin pair as soon as the generators change. aa23004 pins the input instead. a_case_twin_pair_resolves_to_the_install_the_policy_cleared builds the two installs whose keys differ only in case, first asserts the policy selects both, then checks each key resolves to its own install. With the #497 fix reverted and the seed file deleted, it fails with the plan says remove "a", but the deleter resolves that key to "A". The property's doc comment and docs/testing.md now also say that generated cases only take the resolver's exact-match branch. The seed file stays since proptest replays it for free, but nothing relies on it any more.

Comment thread apps/rocm/src/storage.rs Outdated
in_use_key: active.runtime_key.clone(),
folder_survives: active.install_root.is_dir(),
};
let _ = std::fs::remove_dir_all(root);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The temp root is only removed on the success path; an early ? in apply_generated_plan leaks it. A drop guard would clean up on every path.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in aa23004: apply_generated_plan now holds a RemoveOnDrop guard on the test root, so the root is removed on every path, including an early ?.

Comment thread apps/rocm/src/storage/properties.rs Outdated
Some(HoldReason::Marker)
} else if manifest.read_only || manifest.imported_from.is_some() {
Some(HoldReason::NotOwned)
} else if groups[&expected_group(manifest)]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

groups[&expected_group(manifest)] panics with a map-index error when the oracle and the policy disagree about whether an install is held. Use .get() with a prop_assert! so the failure names the install.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in aa23004: it now looks the group up with .get() and fails with a prop_assert! that names the install. Only installs that none of the earlier checks hold reach that branch, and those are exactly the ones recency_groups buckets, so a missing group would mean those checks and held_regardless_of_recency have drifted apart; the assertion now says so rather than panicking.

Comment thread apps/rocm/src/storage/properties.rs Outdated
let mut runner = TestRunner::deterministic();
let draw = (manifests(), retention_inputs());

let labels: [&str; ROWS] = [

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

saw[0..19] is indexed by position against labels and FLOOR; reordering any one of the three silently misaligns them. Named consts or a single struct array would prevent that.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in aa23004: each row's label and floor now live together in one REACH table of (Row, label, floor), saw is indexed by Row::… as usize, and the report first asserts every table row sits at its own Row index. Swapping two rows makes generator_reach_report fail. The measured counts are unchanged.

`rocm storage remove-old-installs` deletes multi-gigabyte install folders
from a set the user never listed, so its safety rules are worth testing
over generated input rather than examples alone.

Property tests for the pure policy (`select_runtimes_to_remove`), 2048
cases each:

- the install in use, the rollback target, anything the active-runtime
  marker names and anything ROCm CLI does not own are never selected,
  including at `--keep 0`;
- each retention group keeps exactly its `--keep` most recent installs
  that nothing else holds (ties broken by key) and removes the rest, and
  raising `--keep` never removes more;
- every install is kept or removed exactly once, and each kept install is
  reported with the first hold reason that applies;
- every selected key resolves, through the real `select_runtime_manifest`,
  to that same install rather than to a case twin of it.

The expected groups and holds are computed in the test from the `--keep`
help and the matching rules the policy implements, not with the policy's
own helpers. Generators draw from small, weighted alphabets so case-twin
keys, shared runtime ids, tied install times, flag/specs disagreement on
the toolchain choice and groups with more candidates than `--keep`
actually occur; a deterministic reach report fails if they stop.

A second property, 256 cases, applies whole plans to real registry files
and install folders through the real removal call and checks the install
in use survives.

The checked-in regression seed is a case-twin input that fails when the
runtime selector fix from #497 is reverted. docs/testing.md gains a
"Removing Old Installs" section listing the commands.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The persisted proptest seed is an RNG seed rather than the input, so a
change to the generators makes it replay a different case. Pin the
case-twin pair the resolver check exists for as an explicit example,
asserting first that the policy selects both twins, and say plainly that
generated cases only exercise the resolver's exact-match precedence.

Also from review:

- add a property that the install the configured default names is never
  selected; dropping only the default hold now fails it at --keep 0;
- check that the whole-plan property removes every install the plan
  named and leaves every other install's folder in place, so neither a
  removal that does nothing nor one that takes a neighbour with it can
  pass, and clean its temp root on every path with a drop guard;
- fail the hold-reason property with a named assertion instead of a map
  index panic;
- keep each reach-report row's label and floor in one table keyed by a
  Row enum, with an order check, instead of three positional arrays;
- update the docs/testing.md section to match and rewrap it.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the test/proptest-storage-retention branch from d5a72f1 to aa23004 Compare November 6, 2026 08:20
@rominf

rominf commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto main and addressed in aa23004.

@juhovainio: there was one rebase conflict, in docs/testing.md, where both sides added a section before "Provider-Assisted Planning"; I kept both. storage:: passes 34/34. select_runtime_manifest now lives in runtime_registry.rs; the tests reach it through the existing re-export, and I re-ran the #497 revert against it. The four inline points are answered in their threads. From the awareness list: the over-long docs/testing.md line is rewrapped, and that section now says to re-measure the floors after any generator change.

On the approving review's follow-ups:

  1. The whole-plan property now also asserts that every install the plan named is gone and every install it left out keeps its folder. It fails if the folder removal is skipped, or if a removal takes a neighbour with it. It still calls uninstall_runtime directly rather than storage()'s loop, so the loop's last-second hold re-check stays covered by prune_removes_the_install_it_named_and_not_a_case_twin_of_it; extracting that loop is a production change I'd rather keep out of this test-only PR.
  2. Added the_install_the_default_names_is_never_selected_for_removal; dropping only the default hold fails it.
  3. The case-twin input is pinned as an explicit example rather than relying on the seed (see the seed-file thread).
  4. Row 19's margin: the counts come from the deterministic RNG, so they only move when the generators or proptest's RNG stream change, and either means re-measuring every floor. docs/testing.md now says so; I kept the uniform four-fifths margin.

On the automated review: the members.len() > 1 gate on rows 12–15 is deliberate. Those rows count a toolchain-reading case only when another recency-decided install in the same group exists for a misreading to affect; row 16 measures the lone-install case. The resolver property's reach is now stated in its doc comment and docs/testing.md, and the whole-plan property checks both directions.

@rominf
rominf requested a review from juhovainio November 9, 2026 08:40

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test and docs only, and the diff matches the description. No blocking correctness issues found; requesting changes for the seed-file inconsistency below, which contradicts the PR's own account.

  1. The committed proptest-regressions/storage/properties.txt contradicts the module docs, the example-test comment and the mutation table, which all say the #497 case is pinned explicitly rather than left to a seed file. The seed is an RNG replay that changes meaning when a generator changes. Drop it or explain why it stays.
  2. The resolver property always takes the exact-key branch (as its own doc admits), so only the pinned example reaches the case-twin path.
  3. The hold-reason oracle mirrors the policy's rule order and case-insensitive pointer handling, so it can agree with the policy by construction. Worth a sentence on why that is acceptable, or an independently stated expectation.
  4. Minor docs nit in docs/testing.md on the storage::properties filter (inline).

CI: all green except E2E tests (Strix Halo, Ubuntu, release, current), which failed at scenario automations-11 (background helper stop), unrelated to a storage-only change; a rerun looks appropriate. I did not build or run the tests, so the mutation table is unverified on my side.

#
# It is recommended to check this file in to source control so that
# everyone who runs the test benefits from these saved cases.
cc 4cab42fe7419af3e985b4da3f1ad9faff893c57b8dc633d49f6c4b87054ca7dd # shrinks to manifests = [InstalledRuntimeManifest { runtime_key: "A", runtime_id: "therock-release:gfx110X-all", channel: "release", format: "wheel", family: "gfx110X-all", family_source: "test", version: "7.0.0", install_root: "rocm-storage-proptest/A", selected_artifact_url: "https://example.invalid/therock", source_layout_generation: None, index_url: None, tarball_file_name: None, python_launcher: None, python_executable: None, pip_cache_dir: None, rocm_sdk: None, sdk_torch: None, wheel_composition: Some(WheelRuntimeComposition { source_layout_generation: "canonical", package_specs: ["rocm[libraries,devel,device-gfx1100]==7.0.0"], rocm_sdk_target: Some("gfx1100") }), read_only: false, imported_from: None, devel: true, installed_at_unix_ms: 0 }, InstalledRuntimeManifest { runtime_key: "a", runtime_id: "therock-release:gfx110X-all", channel: "release", format: "wheel", family: "gfx110X-all", family_source: "test", version: "7.0.0", install_root: "rocm-storage-proptest/a", selected_artifact_url: "https://example.invalid/therock", source_layout_generation: None, index_url: None, tarball_file_name: None, python_launcher: None, python_executable: None, pip_cache_dir: None, rocm_sdk: None, sdk_torch: None, wheel_composition: Some(WheelRuntimeComposition { source_layout_generation: "canonical", package_specs: ["rocm[libraries,devel,device-gfx1100]==7.0.0"], rocm_sdk_target: Some("gfx1100") }), read_only: false, imported_from: None, devel: true, installed_at_unix_ms: 0 }], inputs = RetentionInputs { active_runtime_key: None, previous_runtime_key: None, default_runtime_id: None, marker_runtime_keys: [] }, keep = 0

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new example test and the module docs say the #497 case is pinned explicitly "rather than left to the seed file", and the PR's mutation table runs with the "seed file removed". This seed is an RNG replay that changes meaning whenever a generator changes. Drop it, or say why it stays.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped in fb50ca6. You're right that keeping it contradicted the module's own account. It only held the #497 case as an RNG seed, which is redundant now that the literal example pins that input, and after the file move it would also have been an orphan for main's new proptest_regressions_contract check. The PR body's mutation row now says "no seed file", and both a_case_twin_pair_resolves_to_the_install_the_policy_cleared and the generated resolver property fail with the #497 fix reverted.

/// whose keys differ only in letter case, is pinned as an explicit input in
/// `a_case_twin_pair_resolves_to_the_install_the_policy_cleared`.
#[test]
fn the_deleter_resolves_each_selected_key_to_the_install_the_policy_cleared(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per the doc above, every generated case takes the exact-key branch, so this property cannot reach the case-twin path; only the pinned example does. Consider a generator that emits a case-twin pair deliberately, or fewer cases here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked by mutation, and this turned out to be a wording problem, fixed in fb50ca6. With the seed file gone, I reverted select_runtime_manifest to try the case-insensitive match first (the #497 defect). That fails this generated property as well as the pinned example: the generators produce keys that differ only in case (generator_reach_report's case-twin rows), and under the reverted precedence one of them resolves to its twin. So the property never reaches the fallback, but #497 was about precedence, and the property does reach that. The doc comment, docs/testing.md and the PR body now say this instead of implying only the example covers it. I kept 2048 cases.

Comment thread docs/testing.md Outdated
cargo test -p rocm --bin rocm version_proptests
```

`storage::properties` runs each property at 2048 cases. All but one check the

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

storage::properties also matches the 20,000-sample generator_reach_report and the pinned example test, which are not 2048-case properties; one clause saying so would avoid surprise.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in fb50ca6. The section now says the storage::properties filter also runs generator_reach_report and the pinned example test, which are not properties. It also gives both case counts (2048, and 256 for the whole-plan property) and says PROPTEST_CASES can raise them.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head aa23004f3f82

Outcome: needs work. Test/docs-only change with a sound property-test design, but two CI-contract gaps on the new file plus a currently-failing check need to be resolved before merge.

Blocking

  • New file apps/rocm/src/storage/properties.rs will fail the repo's cargo xtask check-architecture-doc gate (runs in the clippy CI job) — apps/rocm/src/storage/properties.rs, apps/rocm/src/storage.rs:1017
    xtask/src/architecture_doc.rs::is_test_helper only exempts a file whose stem is exactly test_support/test_env/proptests, or ends with _proptests/_props/_properties. This file's stem is bare properties — the exact case the check's own unit test (helper_matching_is_by_whole_file_stem) asserts is not a helper (using the identical-stem example apps/rocm/src/properties.rs). docs/architecture.md has no entry for it, and EXEMPT/EXEMPT_CEILING are both empty. This gate (added on main after this PR's base, and after this PR's last CI run) runs as part of clippy, so it did not catch this yet — but the PR already needs a rebase onto current main (noted elsewhere on this PR), and that rebase will pull the gate in. Fix: rename to match the sibling convention used elsewhere in this codebase (e.g. a flat storage_properties.rs with #[path = "storage_properties.rs"] mod properties;, mirroring therock_version_proptests.rs), or add a citation in docs/architecture.md.
    Confidence 85 · mechanical

  • New proptest case counts are hardcoded, bypassing the project's documented PROPTEST_CASES override contract — apps/rocm/src/storage/properties.rs (cases: 2048), apps/rocm/src/storage.rs (cases: 256, the new applying_a_whole_plan_never_removes_the_install_in_use)
    docs/testing.md's pre-existing "Property tests" section documents that any module setting an explicit case count "reads the variable back itself" through proptest_cases/case_count_from_env (or the uninstall suite's own ROCM_DELETION_PROP_CASES knob), so PROPTEST_CASES=10000 can raise any module's count for a deeper run. Both new proptest blocks use a literal ProptestConfig { cases: N, .. } instead of calling that helper, so they're silently exempt from a documented, universal mechanism. Fix: route both through proptest_cases(2048) / proptest_cases(256) (or a dedicated env knob for the 256-case filesystem one, matching the ROCM_DELETION_PROP_CASES pattern).
    Confidence 80 · mechanical

  • A required-looking CI check is currently failing on this commit — E2E tests (Strix Halo, Ubuntu, release, current)
    The job log shows FAIL: 'automations-helper-stops-when-no-watcher-is-enabled' was expected to pass on this host but FAILED — a regression, exit code 1. This scenario is unrelated to storage/retention, so it reads as a pre-existing flake or host issue rather than something this diff caused — but per CI policy a red check still blocks until it's confirmed green by a rerun.
    Confidence 100 (observed directly in the job log) · the fix is to rerun the job, not to change this PR

Decisions for the author

  • The one resolver-calling property only ever exercises the exact-match branch today — tradeoff
    the_deleter_resolves_each_selected_key_to_the_install_the_policy_cleared calls the real select_runtime_manifest rather than re-deriving its rule (reasonable — there's no way to check "the resolver resolves to itself" without either calling it or duplicating its logic). With today's resolver, every freshly-generated case takes the exact-match branch first, so the case-insensitive fallback this property is nominally probing is reached only by the pinned example test, not by the 2048 generated cases. This is disclosed in the property's own doc comment, so it reads as an accepted, understood tradeoff rather than an oversight — flagging so it's explicitly confirmed as such.

Positive signals

  • The oracle functions (expected_group, expected_default_key, held_regardless_of_recency, installed_the_toolchain) are genuinely independently re-derived from the --keep help text and matching rules rather than thin wrappers around the production helpers — verified line-by-line against retention_group/resolved_default_runtime_key/unconditional_hold/includes_devel.
  • generator_reach_report self-checks that its Row enum and floor table stay in the same order (assert_eq!(*row as usize, index, ...)), closing off the classic silent-misalignment failure mode for this kind of coverage diagnostic.
  • applying_a_whole_plan_never_removes_the_install_in_use checks both directions (nothing planned is left behind, nothing unplanned disappears) against real registry files and real install folders, not just the easier "survivor" direction.
  • docs/testing.md's new section was cross-checked command-by-command and test-name-by-test-name against the actual code and is accurate.

What this covered

Full review of the whole diff (88c344d1…aa23004f): the new apps/rocm/src/storage/properties.rs, the diff to apps/rocm/src/storage.rs, docs/testing.md, and the new proptest-regressions seed file, plus the production code under test (runtime_registry.rs, runtimes_cmd.rs, therock.rs, therock_version_proptests.rs), repo AGENTS.md, and relevant git history (including the PR #497 fix this suite guards). Four independent fan-out passes (core correctness, docs/fixtures, agent-instructions + code comments, history + prior changes) plus a direct read of xtask/src/architecture_doc.rs, docs/testing.md's property-test conventions section, and the PR's live CI checks/logs. No ticket named; description's scope matches the diff.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$(cat /tmp/review_body.md)

Comment thread apps/rocm/src/storage.rs
}

#[cfg(test)]
mod properties;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this new module's file will fail the repo's architecture-doc gate. apps/rocm/src/storage/properties.rs's stem is the bare properties, which xtask/src/architecture_doc.rs's is_test_helper() does not exempt — its own unit test (helper_matching_is_by_whole_file_stem) asserts exactly this stem (via the identical example apps/rocm/src/properties.rs) must not count as a helper. docs/architecture.md has no entry for this file on this PR's head or on current main. The checking file isn't present at this PR's head/merge-base yet (only on main's current tip), so the branch's last CI run never exercised this gate — but it already runs in the clippy job on main and will apply once this branch rebases, which it needs anyway (mergeable_state is already blocked).

Fix: rename to something like storage_properties.rs with #[path = "storage_properties.rs"] mod properties;, mirroring crates/rocm-core/src/process.rs's #[path = "process_tree_properties.rs"], or add a docs/architecture.md citation for the file as it stands.

Confidence 90 · mechanical.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in fb50ca6. The file is now apps/rocm/src/storage_properties.rs, declared from storage.rs as #[path = "storage_properties.rs"] mod properties; (the process_tree_properties.rs pattern), so test paths and the documented storage::properties filter stay the same. I also added a docs/architecture.md row for it, like its sibling property files have. On the branch merged with current main, cargo xtask check-architecture-doc passes.

Comment thread apps/rocm/src/storage/properties.rs Outdated
saw[Row::GroupFamilyTwoWays as usize] |= left.family != right.family;
}
}
if members.len() > 1 {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: these single-manifest reach rows undercount singleton groups. Rows that need a pair (tied install times, case-twin keys, channel/format/family spelled two ways, computed a few lines above this if) are correctly gated behind members.len() > 1. The devel-disagrees, spaced-extra, near-miss-extra and other-package's-spec-first rows below this line are single-manifest .any() checks nested inside the same guard, so a retention group of exactly one install exhibiting any of those conditions is silently excluded from the reach count. No proptest! property is affected — those run over the whole generated group regardless of size — but the measured floor for these four rows under-counts true generator reach.

Fix: move those four .any() checks outside the members.len() > 1 guard, or give them their own ungated block.

Confidence 80 · mechanical.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in fb50ca6. The four single-install toolchain rows (devel flag disagreeing with the specs, spaced extra, near-miss extra, another package's spec first) are now counted for every retention group, including a group of one. A lone install whose toolchain is misread can still move into a sibling group, so the old members.len() > 1 gate undercounted, as you said. I re-measured the floors at four fifths of the new counts (11638 → 9_310, 3427 → 2_740, 3424 → 2_740, 6104 → 4_880). The pairwise rows stay behind the gate.

Comment thread apps/rocm/src/storage/properties.rs Outdated
}

proptest! {
#![proptest_config(ProptestConfig { cases: 2048, ..ProptestConfig::default() })]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this hardcoded case count bypasses the documented PROPTEST_CASES override helper. docs/testing.md's "Property tests" section documents that a module setting an explicit count is expected to read PROPTEST_CASES back itself through the proptest_cases helper. This is a literal cases: 2048 instead. Flagging as non-blocking rather than blocking: the sibling module this one cross-references, apps/rocm/src/therock_version_proptests.rs, already hardcodes cases: 4096/cases: 2048 the same way at six call sites without the helper either, so this follows existing practice in the same area rather than introducing a fresh gap.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in fb50ca6: this block now uses ProptestConfig::with_cases(proptest_cases(2048)), so PROPTEST_CASES can raise it (never lower it). docs/testing.md says so for this suite.

/// day to day is that precedence. The case it exists for, two installs
/// whose keys differ only in letter case, is pinned as an explicit input in
/// `a_case_twin_pair_resolves_to_the_install_the_policy_cleared`.
#[test]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision for the author (tradeoff): this property only ever exercises the exact-match branch today. Calling the real select_runtime_manifest rather than re-deriving its rule is the right call — there's no way to check "the resolver resolves to itself" without either calling it or duplicating its logic. With today's resolver, every freshly generated case takes the exact-match branch first, so the case-insensitive fallback this property nominally probes is reached only by the pinned example test, not by the 2048 generated cases. The doc comment above already discloses this, so it reads as an accepted tradeoff rather than an oversight — flagging so it's explicitly confirmed as such.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed as the intended tradeoff, with a correction in fb50ca6. The property calls the real resolver and so never reaches its case-insensitive fallback. But a resolver that tries that fallback first, the #497 defect, does fail this property on the generated case twins; I checked by reverting it with no seed file present. The doc comment now says it guards the exact-match-first precedence, and that the literal example pins the same input independently of the generators.

Comment thread apps/rocm/src/storage/properties.rs Outdated

/// The input that #497's selector fix is about, pinned here rather than left
/// to the proptest seed file: that file stores an RNG seed, so a change to the
/// generators makes it replay a different case. Two installs whose keys differ

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision for the author (tradeoff): the committed proptest-regressions seed is kept even though this input is now pinned explicitly. This comment already explains why: a seed replays an RNG-generated case rather than a fixed input, so it would replay something else once the generators' shape changes, which is why the #497 case moved to a literal example test instead. The seed file still exists alongside that pinned example. Since the reasoning is already stated here, this reads as a deliberate choice (kept for its own historical replay value) rather than an unexplained inconsistency — worth a one-line confirmation that keeping it serves a purpose beyond habit.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Settled the other way in fb50ca6: the seed file is deleted rather than kept (see the reply on the seed-file thread). The literal example is now the only record of the #497 input, and the comment above it says why a literal is used instead of a seed.

@juhovainio

Copy link
Copy Markdown
Collaborator

Correction: the review above (pull request review #5936587331) was submitted with a tooling error — its body rendered as the literal text $(cat /tmp/review_body.md) instead of the intended report. Posting the actual report here.

Review — needs work

Full review of the whole diff (88c344d1…aa23004f), test and docs only.
No ticket named; the description cites PR #497 only as the reason this suite exists, not as something this PR closes.

Blocking

  • The new file is not exempt from the repo's architecture-doc gate and has no citation — apps/rocm/src/storage/properties.rs (new file), apps/rocm/src/storage.rs:1017, xtask/src/architecture_doc.rs:104-111
    is_test_helper() only exempts a file stem that is exactly test_support/test_env/proptests, or ends with _proptests/_props/_properties. This file's stem is the bare properties, which the check's own unit test (helper_matching_is_by_whole_file_stem) explicitly says must not count as a helper, using the identical-stem example apps/rocm/src/properties.rs. docs/architecture.md has no entry for apps/rocm/src/storage/properties.rs, on this PR's head or on current main. The checking file itself (xtask/src/architecture_doc.rs) isn't present at this PR's head or merge-base yet, only on main's current tip, so the branch's last CI run never exercised this gate — but it already runs in the clippy job on main and will apply once this branch is rebased, which it needs anyway. Sibling precedent for the fix exists: crates/rocm-core/src/process.rs declares #[path = "process_tree_properties.rs"] mod properties;, a stem ending in _properties.
    Confidence 90 · mechanical · Fix: rename to something like apps/rocm/src/storage_properties.rs with #[path = "storage_properties.rs"] mod properties; in storage.rs, or add a docs/architecture.md citation for the file as it stands.

  • A check is currently failing on this commit — E2E tests (Strix Halo, Ubuntu, release, current)
    The job log shows FAIL: 'automations-helper-stops-when-no-watcher-is-enabled' was expected to pass on this host but FAILED — a regression, exit code 1. That scenario is unrelated to storage/retention, and the self-hosted consolidated report for the same run, plus every other self-hosted hardware lane, came back green — so this reads as a pre-existing flake rather than something this diff caused. Still, a red check blocks until a rerun confirms it.
    Confidence 100 (read directly from the job log) · fix is to rerun the job, not to change this PR.

Non-blocking

  • generator_reach_report's single-manifest rows undercount singleton groups — apps/rocm/src/storage/properties.rs:806-825
    Rows measuring tied install times, case-twin keys, and channel/format/family spelled two ways are pairwise checks correctly gated behind members.len() > 1 (you need two members to form a pair). The devel-disagrees, spaced-extra, near-miss-extra and other-package's-spec-first rows are single-manifest .any() checks nested inside that same if members.len() > 1 block, so a retention group of exactly one install exhibiting any of those conditions is silently excluded from the reach count. No proptest! property is affected (those run over the whole generated group regardless of size); only the measured floor for those four rows under-counts true generator reach.
    Confidence 80 · mechanical · Fix: move those four .any() checks outside the members.len() > 1 guard, or give them their own ungated block.

  • The new proptest case counts bypass the documented PROPTEST_CASES override helper — apps/rocm/src/storage/properties.rs:384
    docs/testing.md's "Property tests" section documents that a module setting an explicit case count is expected to read PROPTEST_CASES back itself through the proptest_cases helper, so PROPTEST_CASES=10000 can deepen any module's run. This module's ProptestConfig { cases: 2048, .. } is a literal instead. Noted as non-blocking because it isn't a new pattern: the sibling module this one cross-references, apps/rocm/src/therock_version_proptests.rs, already hardcodes cases: 4096/cases: 2048 the same way at six call sites without using the helper either, so this file follows existing practice in the same area rather than introducing a fresh gap.

Decisions for the author

  • The one resolver-calling property only ever exercises the exact-match branch today — tradeoff
    the_deleter_resolves_each_selected_key_to_the_install_the_policy_cleared (apps/rocm/src/storage/properties.rs:580) calls the real select_runtime_manifest rather than re-deriving its rule, which is the right call — there's no way to check "the resolver resolves to itself" without either calling it or duplicating its logic. With today's resolver, every freshly generated case takes the exact-match branch first, so the case-insensitive fallback this property is nominally probing is reached only by the pinned example test, not by the 2048 generated cases. The property's own doc comment already discloses this, so it reads as an accepted tradeoff rather than an oversight.

  • The committed proptest-regressions seed is kept though the fix(runtimes): resolve a runtime selector to the key that matches exactly #497 case is pinned explicitly — tradeoff
    apps/rocm/proptest-regressions/storage/properties.txt replays an RNG-seeded case, and apps/rocm/src/storage/properties.rs:694-699's own comment explains why the fix(runtimes): resolve a runtime selector to the key that matches exactly #497 input is pinned as a literal example test instead of relying on that file: a seed replays a different case once the generators change shape. The seed file still exists alongside that pinned example. The code already states the reasoning, so this reads as a deliberate choice (kept for its own historical replay value) rather than an unexplained inconsistency — worth a one-line confirmation that keeping it serves a purpose beyond habit.

Positive signals

  • The oracle functions (expected_group, expected_default_key, held_regardless_of_recency, installed_the_toolchain) are independently re-derived from the --keep help text and the policy's documented matching rules rather than thin wrappers around the production helpers they check.
  • generator_reach_report asserts its Row enum and its floor table stay in the same order (assert_eq!(*row as usize, index, ...)), closing off a silent-misalignment failure mode for this kind of coverage diagnostic.
  • applying_a_whole_plan_never_removes_the_install_in_use checks both directions against real registry files and real install folders: nothing the plan named is left behind, and nothing the plan left out disappears.
  • apply_generated_plan's RemoveOnDrop guard cleans up its scratch tree even when an early ? returns partway through a case.

Deployment notes

None — test and documentation only, no dependency, migration or behavior change.

What this covered

Read completely: the new apps/rocm/src/storage/properties.rs (979 lines), the diff to apps/rocm/src/storage.rs (the new mod properties; declaration and the new AppliedPlan/apply_generated_plan/applying_a_whole_plan_never_removes_the_install_in_use test code), the docs/testing.md diff, and the new apps/rocm/proptest-regressions/storage/properties.txt. Verified the architecture-doc finding directly: read xtask/src/architecture_doc.rs in full including its own unit tests, confirmed via git cat-file/git show that file is absent at this PR's head and merge-base but present on current main, confirmed docs/architecture.md has no citation for the new file on either ref, found the sibling _properties-suffixed precedent in crates/rocm-core/src/process.rs, and cross-checked the project's own contribution guide, which states the same rule in its architecture-guardrails section. Checked live status via the PR's check-run rollup for this exact commit and read the one failing job's log directly rather than assuming its cause. No Rust toolchain was available in this environment, so nothing was compiled or executed; the PR's own mutation-evidence table and "34/34 passing" claim are taken from the description, not independently run.

Four earlier reviews exist at or near this head, surfaced by a required duplicate-review check before this one started, so this review did not begin fully blind of their content; every point reconciled above (the architecture-doc gate, the reach-report gating, the case-count convention, the two tradeoffs) was independently re-derived and verified against the source, CI logs, and git history rather than taken on their word, and several of their already-raised points (a temp-directory leak on early return, a possible panic on a missing map entry, the reach-report's row/label/floor ordering, and an assertion that only checked one direction of removal) read as already fixed in this exact commit. Not reconciled against any PR discussion thread beyond the review bodies themselves.


🤖 Automated by an AI agent

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change (test/docs-only; no production code touched). No ticket named; the description references PR #497 as prior context for the bug this suite guards, not as a ticket this PR closes. Description's scope ("test and docs only; no production code changes") matches the diff.

Blocking

  • A CI check is currently red on this commit — E2E tests (Strix Halo, Ubuntu, release, current)
    The job log for this exact head (aa23004f) shows FAIL: 'automations-helper-stops-when-no-watcher-is-enabled' was expected to pass on this host but FAILED — a regression, exit code 1. The failing scenario is an automations/watcher-stop check, unrelated to storage/retention, so it reads as a pre-existing flake or host issue rather than something this diff caused — but it is still red today.
    Confidence 100 (read directly from the job log) · Fix: rerun the job to confirm green before merge.

Non-blocking

  • Both new proptest suites hardcode a literal case count instead of the crate's proptest_cases() env-override helper — apps/rocm/src/storage.rs:1960, apps/rocm/src/storage/properties.rs:384
    docs/testing.md's pre-existing "Property tests" section documents that a module setting an explicit count "reads the variable back itself" through the proptest_cases helper (apps/rocm/src/test_support.rs:61), used by gpu_selection_proptests.rs, providers/local_http/proptests.rs, providers/wire/sse_emitter_proptests.rs and crates/rocm-core/src/examine_driver_proptests.rs. Both of this PR's new blocks use a bare ProptestConfig { cases: N, .. } instead, so PROPTEST_CASES=10000 — the exact command this PR's own new docs section recommends for other properties — silently has no effect on either new suite. This isn't a deviation unique to this PR, though: the pre-existing, untouched apps/rocm/src/therock_version_proptests.rs (six blocks, cases: 2048/4096) already uses the identical bare-literal pattern, so the docs' "all of them" framing was already inaccurate before this change, and this PR extends an existing (if undocumented) precedent rather than inventing a new one.
    Confidence 80 · mechanical · Fix: route both through proptest_cases(2048)/proptest_cases(256) for consistency with the helper-based suites, or note the literal-config pattern as a second accepted style in docs/testing.md.

Decisions for the author

  • The architecture-doc gate doesn't exist in this PR's history today, but will apply on the next rebase — tradeoff
    xtask/src/architecture_doc.rs is absent at both this PR's base and its tip (confirmed directly); it was added to main after this PR diverged. git merge-tree between current origin/main and this PR's tip resolves with no conflicts, so a rebase today would bring the checker in alongside the new apps/rocm/src/storage/properties.rs. That checker's is_test_helper matches by whole file stem, and its own unit test explicitly treats a bare properties.rs stem as not exempt — this file's stem is exactly properties, so it will need a docs/architecture.md citation or a rename (e.g. storage_properties.rs) to pass post-rebase. Flagging so it's handled at rebase time rather than discovered by a red clippy job afterward.
  • The one property that calls the real resolver only ever exercises its exact-match branch on freshly generated cases — tradeoff
    the_deleter_resolves_each_selected_key_to_the_install_the_policy_cleared calls the real select_runtime_manifest (verified against the actual #497 fix commit: it matches runtime_key == selector before any case-insensitive fallback), and manifests() only dedupes exact keys, so the case-insensitive branch this property is nominally about is reached only by the hand-pinned a_case_twin_pair_resolves_to_the_install_the_policy_cleared, not by the 2048 generated cases. This is disclosed accurately in both the property's own doc comment and the PR description, so it reads as an understood, deliberate tradeoff rather than an oversight.

Positive signals

  • The oracle functions in storage/properties.rs (expected_default_key, held_regardless_of_recency, installed_the_toolchain, expected_group) are genuine independent reimplementations of resolved_default_runtime_key/unconditional_hold/includes_devel/retention_group, traced line-by-line and found faithful, including the case-insensitive-pointer and devel-flag/package-spec-disagreement rules.
  • generator_reach_report self-checks its own Row enum against its floor table's array order (assert_eq!(*row as usize, index, ...)), closing off the classic silent-misalignment failure mode for this kind of diagnostic.
  • RemoveOnDrop (storage.rs) cleans up its scratch tree on every exit path, including an early ? — a real improvement over the pre-existing hand-written tests in the same file, which only remove their root as the last line and leak it if an earlier assertion aborts.
  • applying_a_whole_plan_never_removes_the_install_in_use checks both directions (nothing planned is left behind, nothing unplanned disappears) against real registry files and real folders, and its doc comment accurately discloses what it does not re-check (the production loop's last-second hold re-check, left to the existing prune_removes_the_install_it_named_and_not_a_case_twin_of_it).
  • docs/testing.md's new section gives exact, runnable commands and accurately states the case counts actually present in the code.

Deployment notes

None — test and documentation only, no production code changed.

What this covered

Full review of the whole diff (88c344d1…aa23004f): fanned out two independent passes — one reading the new apps/rocm/src/storage/properties.rs (979 lines) completely against its production oracle (runtime_registry.rs, therock.rs) and the regression seed file; one reading the diff and the full apps/rocm/src/storage.rs (2026 lines) against runtime_uninstall.rs and select_runtime_manifest's contract in runtime_registry.rs — plus a third pass covering docs/testing.md's diff and its pre-existing Property-tests section, the root AGENTS.md, git history of the #497 fix this suite guards, this commit's live CI check list and the one failing job's log, a direct read of xtask/src/architecture_doc.rs and its own unit test, and a conflict-free git merge-tree check between this PR's tip and current origin/main.

Did not run cargo test/cargo xtask coverage (no Rust toolchain in this environment); the PR's own "34/34 passing" claim and mutation-evidence table are read from the PR description and from CI's green build-and-test/Test (affected crates)/clippy results, taken at face value rather than independently executed.

Not reconciled against any prior review or discussion on this PR; this is a blind pass, cross-checked against the code, git history (including the actual #497 fix commit) and a live git merge-tree, not against anyone's earlier conclusions.


🤖 Automated by an AI agent

Comment thread apps/rocm/src/storage.rs Outdated
// Each case writes and deletes a tree, so this matches the other
// filesystem-backed property suites rather than the pure-policy
// budget.
cases: 256,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this hardcodes cases: 256 instead of going through the crate's proptest_cases() helper (apps/rocm/src/test_support.rs:61), which every other env-scalable proptest suite in the crate uses (gpu_selection_proptests.rs, providers/local_http/proptests.rs, providers/wire/sse_emitter_proptests.rs, crates/rocm-core/src/examine_driver_proptests.rs). docs/testing.md's "Property tests" section documents this as universal, so PROPTEST_CASES=10000 silently has no effect on this suite.

That said, this isn't a novel deviation: the pre-existing, untouched apps/rocm/src/therock_version_proptests.rs already uses the same bare ProptestConfig { cases: N, .. } pattern (six blocks), so the docs' "all of them" framing was already inaccurate before this PR. Worth routing through proptest_cases(256) for consistency, or acknowledging the literal-config pattern as a second accepted style in the docs.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in fb50ca6: ProptestConfig::with_cases(crate::test_support::proptest_cases(256)). I left therock_version_proptests.rs's bare literals for a separate change, since that file is outside this PR.

Comment thread apps/rocm/src/storage/properties.rs Outdated
}

proptest! {
#![proptest_config(ProptestConfig { cases: 2048, ..ProptestConfig::default() })]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same pattern as the new storage.rs property (see that inline comment): cases: 2048 is a bare literal rather than proptest_cases(2048), so PROPTEST_CASES doesn't scale this suite either. Non-blocking given the pre-existing therock_version_proptests.rs precedent for the same bare-literal style, but worth aligning with the helper-based suites for consistency.

Separately (tradeoff, not a defect): xtask/src/architecture_doc.rs doesn't exist anywhere in this PR's history today (confirmed absent at both base and tip), so this file isn't currently gated — but it was added to main after this PR's base, and git merge-tree shows this branch merges cleanly with current main, so the next rebase will pull that checker in. Its is_test_helper matches by whole file stem, and its own unit test explicitly treats a bare properties.rs stem as not exempt — this file's stem is exactly properties, so it will need either a docs/architecture.md citation or a rename (e.g. storage_properties.rs) to pass post-rebase.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both points are fixed in fb50ca6: the case count goes through proptest_cases(2048), and the file is renamed to storage_properties.rs with an architecture-doc row. cargo xtask check-architecture-doc passes on the branch merged with current main. See the thread on storage.rs for details.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…ew points

Main gained an architecture-doc check that exempts property-test files only
by a `_properties`-style stem, so the bare `storage/properties.rs` would fail
it. Move it to `storage_properties.rs`, declared through `#[path]` as the same
`properties` module (as `process_tree_properties.rs` is), so test paths and
the documented `storage::properties` filter are unchanged, and give it an
architecture row like its sibling property files.

Drop the saved proptest seed instead of carrying it to the new name. It held
the #497 case-twin failure as an RNG seed, which replays a different input
once the generators change; that input is pinned as the literal example
`a_case_twin_pair_resolves_to_the_install_the_policy_cleared`, so the seed was
redundant, and left in place it would also be orphaned by the move.

Read both suites' case counts through `proptest_cases`, so `PROPTEST_CASES`
can raise them as it does for the other suites.

Count the four single-install toolchain rows of `generator_reach_report` in
groups of one too: a lone install whose toolchain is misread can still move
into a sibling group. Floors re-measured at four fifths of the new counts.

Correct what the resolver property is said to guard: generated case twins do
fail it when the resolver tries a case-insensitive match first (checked by
reverting that precedence), so it is not only the pinned example that covers
#497. State why the hold-reason oracle follows the policy's own order, and
say which tests the `storage::properties` filter runs besides properties.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@github-actions

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants