Skip to content

refactor(rocm): split therock.rs into child modules, part 1 of 3: registry, version, io, process, fetch - #824

Merged
jussielo-amd merged 7 commits into
ROCm:mainfrom
jussielo-amd:refactor-therock-split-foundation
Nov 2, 2026
Merged

jussielo-amd merged 7 commits into
ROCm:mainfrom
jussielo-amd:refactor-therock-split-foundation

Conversation

@jussielo-amd

Copy link
Copy Markdown
Collaborator

Why

apps/rocm/src/therock.rs is the largest file in the repository, about 12,600 lines with roughly 7,000 of production code in sixteen clusters that change for different reasons, and it sits in a module cycle: therock called runtimes_cmd for the active-runtime lookup while runtimes_cmd imported therock. This is the first of three PRs that split it into child modules behind a parent that re-exports what other files use. It lands the part every later move depends on: a crate-level runtime_registry.rs that ends the production dependency of therock on runtimes_cmd, and the four leaf modules (version, io, process, fetch) that the remaining clusters build on. Delivers ROCMAI-799, ROCMAI-800, ROCMAI-801, ROCMAI-802 and ROCMAI-803.

What changes

Five commits, one per step, in dependency order so the branch can be reviewed commit by commit, plus a final commit that corrects documentation:

  1. runtime_registry.rs takes the runtime manifest model, its loaders and the SDK probe validators out of therock.rs.
  2. The registry selection and activation validation move there from runtimes_cmd.rs, the duplicate path helpers in runtimes_cmd.rs are deleted, and therock's consent code now reaches the registry through crate::runtime_registry. That removes the production edge from therock to runtimes_cmd.
  3. therock/version.rs takes the version grammar and build-date helpers.
  4. therock/io.rs takes the scratch-directory and atomic-write helpers and therock/process.rs takes the subprocess, progress and tarball-extraction helpers. PROCESS_ENV_TEST_LOCK moves to test_support.rs as one static, so tests in every child still serialise on the same lock.
  5. therock/fetch.rs takes the signed-metadata cache, downloads and the pinned key constant, and the verify_pinned_keys path pin follows it.

The 14 files that import from therock are unchanged except main.rs, which gains the mod runtime_registry; line and drops one import that became unused. The serde shape of the registry manifest structs is byte-identical to the original, so the on-disk format does not change.

Besides the relocations, the commits make these edits, which the first five commit messages understate: helpers that other modules use are widened to pub(super) or pub(crate) (including the fields of ParsedVersion and VersionOrderKey that the property tests read); runtime_version_build_date is narrowed from pub(crate) to pub(super); mod io shadows std::io in therock.rs, so three io:: paths are now std::io::; the moved tests use small copies of the test_paths fixture in each new test module; two doc links are now super:: paths; comment pins in xtask and the e2e harness name the new files; and each new file carries a module doc and an entry in docs/architecture.md. The final commit corrects statements in those new docs that the review found inaccurate (which module depends on which, what the atomic writers are used for, and what is pub(super)).

Verification

  • cargo fmt --check is clean, and cargo clippy -D warnings is clean on rocm at every commit and on xtask at the final head.
  • cargo test -p rocm --bin rocm -- --list gives the same 1,374 tests and 7 ignored tests before and after, with identical leaf names, including the 25 version_proptests and the two archive_properties cases.
  • cargo test -p rocm --bin rocm -- therock:: runtime_registry:: runtimes_cmd:: passes 257 tests, and the xtask env_mutation_contract, ascii_output_contract and verify_pinned_keys tests pass.
  • cargo xtask manifest --check and cargo xtask check-crate-edges pass.
  • The verify_pinned_keys entry is dormant today (no canonical metadata key is published), so a stale path would not fail CI. The new path was checked separately with a temporary key file, which reported the constant found in therock/fetch.rs.
  • Not run locally: the full workspace tests, the tests/e2e-cucumber suite and the Windows lane. The cfg(windows) code that moved (publish_temp_file and replace_file_windows in io.rs, the Windows temp-directory handling in process.rs) is exercised only on the Windows lane; a compile check with the cfg gates forced showed no unused or missing imports, but that is not a Windows build.

Scope

This is an internal refactor with no user-observable change, so no end-to-end scenario is added; the unmodified tests/e2e-cucumber suite covers the unchanged behaviour. The remaining clusters (source and family resolution, managed Python, probes, tarball and wheel resolution, the consent gate and update planning) follow in the next two PRs and are not part of this one.

Template checklist: not a bug fix, so there are no stale xfail rows to narrow; no new subcommand or subsystem; no user-facing message is added or changed.

Create a crate-level runtime_registry module holding the installed-runtime
manifest model, the registry path helpers, the manifest loaders and writer,
the wheel-composition readers and the rocm_sdk probe validator. therock
re-exports the names other modules reach as therock::X, so no consumer
changes. The probe validator moves here rather than to a therock child so
that the manifest validators can call it without a new crate-level cycle.

Pure move: the serde shape of the registry files is unchanged, and the
tests that exercise only the moved items move with them.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Move runtime selection, usability and activation validation out of
runtimes_cmd into runtime_registry, and delete runtimes_cmd's duplicate
registry path helpers (their bodies were identical to the ones that moved
there in the previous commit). runtimes_cmd re-exports the names other
modules import from it, so those importers are unchanged. therock's consent
code now reaches the registry through crate::runtime_registry, which ends
the therock to runtimes_cmd dependency.

Pure move; the selector test moves with the code.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Move the version parsing, PEP 440 ordering, host-version reader and
build-date helpers out of therock.rs into a leaf child module. therock
re-exports runtime_version_display for the modules that render versions;
the version property tests now import the moved items from the new module.

Pure move; the one test that exercises only the moved items moves with them.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…therock/process.rs

Move the scratch-directory and atomic-write helpers into a leaf io module,
and the command-running, bounded-wait, progress-line and tarball-extraction
helpers into a process module that depends on it. Extraction lives in
process so that io and process do not depend on each other. therock
re-exports write_file_atomically for its other callers.

The process-wide environment lock the tests share moves out of the therock
test module into test_support as a single static, so the tests that stay in
therock and the moved ones still serialize on the same lock. The archive
property tests move under process.

Pure move; the tests that exercise only the moved items move with them.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Move the metadata GET with its etag cache and detached-signature
verification, the streaming artifact download, and the tarball size and
free-space preflight into a child module that depends on the io module.
therock re-exports truthy_env, which the driver install code reads.

The pinned metadata key constant moves with it, so the path the
verify-pinned-keys check reads (and the two doc lines that name it) now
point at the new file.

Pure move; the tests that exercise only the moved items move with them.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Comment and documentation fixes for statements in the five split
commits that were inaccurate or stale. No code changes.

- runtime_registry module doc and architecture entry: say the module
  also holds registry selection and activation validation, that it
  depends on neither therock nor runtimes_cmd, and that runtimes_cmd
  still reaches the manifest model through therock's re-exports while
  therock no longer depends on runtimes_cmd in production code. They
  used to say the two modules no longer depend on each other.
- therock/io.rs module doc and architecture entry: the atomic writes
  serve the metadata cache entries, the staged metadata public key, the
  startup update-check record and the active-runtime marker, not the
  runtime registry manifests, which are written with a plain fs::write.
- architecture entry for therock/fetch.rs: list the pub(super) items,
  including CachedHttpText and its text field, instead of saying
  everything else is private.
- architecture entry for therock/version.rs: the helpers behind the
  pub(super) items are private, not all items pub(super).
- test_support: the PROCESS_ENV_TEST_LOCK doc names the therock tests
  and the variables it covers, and notes that the main.rs test env
  guard and the engine_ipc production lock are separate.
- VersionOrderKey doc: release and stage are pub(super) because the
  property tests read them, so it no longer claims no fields are
  reachable.
- Comments in tests that stayed in therock.rs now point at the guard
  tests in therock::process and the HTTP servers in the fetch tests,
  and say the shared env lock lives in test_support.
- The proptests module doc and the comparator doc say where the
  comparators live now; an e2e harness comment names
  therock/process.rs as the extraction site.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new header mention causes the extractor to read Remaining Owner Step instead of the pinned PEM, so verification will fail once the canonical metadata key is published.

1 open issue
What changed in this PR

This first stage of the TheRock refactor separates runtime registry responsibilities and install helpers, removing the production dependency from therock to runtimes_cmd while preserving existing caller interfaces and manifest formats.

Changes:

  • Extract runtime manifests, selection, and validation into runtime_registry.
  • Move version, filesystem, subprocess, and fetch helpers into child modules, retaining tests and a shared environment lock.
  • Update architecture documentation, source-reference comments, and the pinned-key checker’s source path.
File Description
xtask/​src/​verify_pinned_keys.rs Points metadata-key verification to therock/fetch.rs.
xtask/​src/​main.rs Updates the key-verification help comment.
xtask/​src/​env_mutation_contract.rs Updates the shared-lock location in documentation.
xtask/​src/​e2e_prewarm.rs Updates the version-helper reference comment.
tests/​e2e-cucumber/​tests/​e2e/​comfyui_steps.rs Updates the probe-validator reference comment.
tests/​e2e-cucumber/​tests/​e2e.rs Updates the extraction reference comment.
docs/​architecture.md Documents module ownership and dependencies.
apps/​rocm/​src/​therock/​version.rs Houses version parsing, ordering, and date helpers.
apps/​rocm/​src/​therock/​process.rs Houses subprocess, progress, and extraction helpers and tests.
apps/​rocm/​src/​therock/​io.rs Houses temporary-directory and atomic-write helpers and tests.
apps/​rocm/​src/​therock/​fetch.rs Houses downloads, signed metadata caching, and related tests.
apps/​rocm/​src/​therock_version_proptests.rs Adjusts imports and documentation for relocated helpers.
apps/​rocm/​src/​test_support.rs Centralizes the existing TheRock environment-test lock.
apps/​rocm/​src/​runtimes_cmd.rs Re-exports relocated registry helpers and removes duplicate paths.
apps/​rocm/​src/​runtime_registry.rs Owns manifest models, persistence, selection, and validation.
apps/​rocm/​src/​main.rs Declares the registry module and removes an unused import.

Review effort: Balanced

Comment thread apps/rocm/src/therock/fetch.rs Resolved
@jussielo-amd
jussielo-amd marked this pull request as ready for review November 2, 2026 10:54
@jussielo-amd
jussielo-amd requested a review from a team as a code owner November 2, 2026 10:54
Remove redundant text from documentation comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@jussielo-amd
jussielo-amd force-pushed the refactor-therock-split-foundation branch from 5aa28ca to 2119ee1 Compare November 2, 2026 11:04
@jussielo-amd
jussielo-amd requested a balanced review from Copilot November 2, 2026 11:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the runtime-registry and helper-module extraction.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · 2119ee1

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (7 commits, 17 files, +4179/-3871).
Delivers ROCMAI-799, ROCMAI-800, ROCMAI-801, ROCMAI-802, ROCMAI-803 (Jira sub-tasks for Phase 8.1-A steps 1-5, all currently "In Progress"). The PR explicitly scopes itself as part 1 of 3 of a larger therock.rs split; the description's claimed scope matches the actual diff, and none of the ticket summaries are claimed as closed.

Blocking

None

Non-blocking

None

Decisions for the author

None

Positive signals

  • The seven commits are genuinely atomic and reviewable in isolation (one extraction step per commit, plus a separate doc-correction commit) — this let each independent review pass diff a moved function against its pre-move source byte-for-byte rather than trusting the "pure move" framing, and every slice held up under that check.
  • Consolidating PROCESS_ENV_TEST_LOCK into a single pub(crate) static in test_support.rs (apps/rocm/src/test_support.rs:33) is the correct fix for the real hazard a naive split would create: every env-mutating/bare-spawn test across therock.rs, therock/io.rs, and therock/process.rs was confirmed to still serialize on the one lock instance, so the split introduces no test-isolation race.
  • The final commit (9127618) demonstrably fixed real doc/comment inaccuracies left by earlier commits (confirmed by diffing intermediate commit states against final head) — evidence the author re-verified doc claims against code rather than writing them once and moving on.

Deployment notes

  • No user-observable behavior change is claimed or found; the registry manifest structs' serde shape was independently confirmed byte-identical (field names, #[serde(default...)] attributes, variant names all unchanged).
  • At review time, CI's E2E tests and windows-build-and-test jobs were still in_progress; every check that had completed (clippy, build-and-test, Test (affected crates), Coverage, license/notices, docs-links, commit-signatures, CodeQL, etc.) passed. The cfg(windows) code that moved (publish_temp_file/replace_file_windows in io.rs, Windows temp-handling in process.rs) is only exercised by the Windows lane, which the PR text itself flags as not run locally — worth confirming windows-build-and-test lands green before merge.
  • This PR is explicitly the foundation for two follow-up PRs; nothing in it should need to be re-opened once merged, since the remaining therock.rs clusters are untouched here.

What this covered

Every one of the 5 new/changed modules (runtime_registry.rs, therock/version.rs, therock/io.rs, therock/process.rs, therock/fetch.rs) read completely at pr-824-head, each diffed line-by-line against its pre-move source in therock.rs on origin/main (11a6069…2119ee1), via 5 independent fan-out workers grouped by commit/coupling: (1) registry extraction (e38d25a, 5bd8fce), (2) version extraction (38bec5d), (3) io/process extraction (d5f5d58), (4) fetch + pinned-key extraction (4ab2cf1, 2119ee1), (5) the doc-correction commit (9127618) plus docs/architecture.md and cross-module wiring in the surviving therock.rs. Each worker also checked the small incidental comment-only diffs in xtask/ and tests/e2e-cucumber/. I additionally: read the repo's agent-instruction file (AGENTS.md) in full and checked the diff against its doc-link and architecture-guardrail conventions (no violations found); independently re-derived the pinned-key-extraction logic in xtask/src/verify_pinned_keys.rs myself to resolve a conflict with an earlier automated bot comment flagging a token-ordering risk there — confirmed no such risk exists in the code as written (only one occurrence of the token in fetch.rs, at the real const declaration), consistent with that same bot's own follow-up review at this head reporting the issue resolved; and retrieved the 5 named Jira tickets to confirm the description's scope claim.

Did not run: a dedicated "prior review comments" pass beyond what the mandatory pre-review claim-check surfaced (an automated bot's comments, read only for that check and independently re-verified rather than trusted, per method); a local cargo build/test/clippy run of my own (relied on the PR's disclosed verification matrix plus live CI results instead, consistent with reading what CI already reports rather than re-deriving it). Initial attempts to launch the fan-out hit a shared environment-wide concurrency limit and had to be retried in sequence; all 5 workers eventually launched and completed with full independent coverage, so this delayed but did not reduce coverage. Not reconciled against any PR discussion beyond the mandatory head-commit claim-check.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I compared the removed and added lines across the whole diff, and the move is verbatim. The only differences are visibility changes (pub(super)/pub(crate)), path re-points to the new modules, and doc/comment re-pointing. No logic changed, and the test count is the same before and after (43 test attributes removed, 43 added).

  • The five steps are all there with no scope creep, and the module docs and docs/architecture.md match the new layout.
  • CI is green, including the Windows build/test, E2E and clippy.
  • The visibility widening is minimal and disclosed in the description.
  • The test_paths fixture is duplicated across the new test modules. That reads as deliberate, so I'm leaving it as is.

This is a diff-level review; I didn't run the build locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants