Repository navigation
EAI-7404: make runtime activation and rollback transactional - #424
Conversation
847a013 to
f2984db
Compare
d7da211 to
40666ee
Compare
nowycondro
left a comment
There was a problem hiding this comment.
Summary
Change type: bug fix + feature — makes rocm runtimes activate / rollback transactional across its two persisted writes, and adds live-service reconciliation with an opt-in --restart-services.
Overall assessment: Needs work. The transaction half (snapshot / restore / atomic writes) is careful and I could not break it. The service-reconciliation half has one real correctness gap, and the rocm install sdk path pays this PR's new failure mode without receiving any of its benefit.
Reviewed against AGENTS.md. No Rust toolchain exists in this sandbox, so nothing below was compiled or executed — findings are static, traced through source. CI is the authority on build/test, and CI is green (see the note on the PR body at the end).
Blocking
1. The "env-pinned services are never moved" guarantee does not hold for any live vLLM service
apps/rocm/src/main.rs:8278
None if record.env_id.is_some() => ServiceRuntimeState::Matches,The arm fires only when record.runtime_id is None. For a real service that state does not survive first contact with the engine:
resolve_engine_selection(main.rs:21388) setsruntime_id: None, env_id: Some(..)for a--env-idlaunch, so the record starts out matching the arm.- vLLM's
write_running_state(engines/vllm/src/lib.rs:2015) writes"runtime_id": runtime.runtime_idinto the engine state file.VllmRuntime.runtime_idis a plainString, not anOption(engines/vllm/src/lib.rs:132), and every construction site fills it — falling back to literals like"external-vllm"(lib.rs:1166) when nothing else resolves. It is never empty. refresh_from_engine_state(crates/rocm-core/src/lib.rs:7740) then doesself.runtime_id = Some(..)from that value. It never clears the field.
So the moment an env-pinned vLLM server writes its state file, record.runtime_id becomes Some(..), the match takes the Some(recorded) arm, and the service classifies as Stale.
Failure scenario. User runs rocm serve --env-id my-env, then rocm runtimes activate <other-runtime> --restart-services --yes. The env-pinned server is reported as left behind, stopped, and re-pinned — and pin_service_record_to_runtime (main.rs:8540) deliberately clears env_id, so the user's explicit environment pin is destroyed. The doc comment at main.rs:8237-8240, the README, and the PR body all promise the opposite.
Why this was not caught: runtime_activation_ignores_self_managed_engine_services (main.rs:31985) plants svc-env-pinned and asserts it is not stale. It passes because plant_service_record_on_runtime (main.rs:27669) writes an engine state file containing only {"status": ...} — no runtime_id key — so refresh_from_engine_state leaves runtime_id at None. No real engine writes a state file that sparse. The test asserts the arm, not the behaviour.
Note the code's own doc comment is precise — "pinned to an explicit env_id with no recorded runtime" — while the PR body and README drop that qualifier. The comment describes what the code does; the user-facing text describes what it was meant to do. Those have diverged.
Suggested fix. Decide what an env pin means and make one thing true everywhere. Either (a) treat env_id.is_some() as never-stale regardless of runtime_id, moving the guard above the runtime_id match — but that reintroduces the problem reviewer-focus (a) identified, since vLLM sets env_id on every launch, so it would exclude everything; or (b) accept that env_id is not a user pin (which is what round 1 concluded) and correct the doc comment, the README and the PR body to say so, dropping the now-unreachable arm. (b) looks right and is consistent with the rest of the PR. Either way, give the test a fixture whose engine state resembles what vLLM actually writes, or it will keep passing whichever answer is chosen.
2. rocm install sdk runs the reconciliation, inherits its new failure mode, and throws the result away
apps/rocm/src/main.rs:11224, 11232, 11193
finalize_successful_sdk_install calls activate_runtime twice and discards activation.services both times — once via let _ =, once by building a SdkInstallFinalization that has no services field. render_sdk_install_success never prints a reconciliation section.
This is the direct answer to your reviewer-focus (d). The "fail clean rather than half-apply" trade is defensible on activate / rollback / update --apply --activate, because the user sees the report that the reconciliation produced. On install sdk the user sees nothing, so the path pays the entire cost — an unreadable services folder can now fail the finalization of a first-run SDK install, plus per-service liveness and inference probe latency — for zero benefit.
It is also contrary to the stated intent right next door: the ROLLBACK_RECOVERY_HINT doc comment (main.rs:18432) says every activation path, "install sdk" named explicitly, "gives the same advice" and lists render_sdk_install_success as a call site. Service reconciliation is shown on two of the three.
Suggested fix. Either carry services into SdkInstallFinalization and render it (restores parity, keeps the failure mode justified), or give activate_runtime a way to skip reconciliation for this path so a fresh install cannot be failed by unrelated service-directory I/O. The first is more consistent; the second is safer for a setup path.
3. No automated coverage of the --restart-services --yes success path
AGENTS.md §3: "User-observable behavior needs a scenario, not only a unit test... a unit test asserting the internal helper does NOT discharge this." The escape hatch it offers is a gated CI lane, named in the PR text.
restart_stale_runtime_services is exercised only through its failure branches (pre-stop refusal, post-stop failure/restore). Nothing — unit, mock e2e, or gated lane — reaches restart_service_onto_runtime returning Ok(()). The observable behaviour with no automated coverage: the services_restarted: line, the restarted service dropping off services_on_previous_runtime, and exit 0.
The PR discloses this honestly and points at docs/manual-testing.md. A manual doc is not "another CI level", so as written it does not discharge §3 — though a maintainer may reasonably accept the disclosure given a real restart needs a spawned process. Flagging it so that acceptance is a decision rather than an oversight.
Non-blocking
-
config.saveruns twice on the update path.main.rs:18417-18418:activate_runtimealready persisted viasave_activated_config, thenconfig.save(paths)?runs again with no mutation in between. A failure there reports an error after an activation that fully succeeded, sending the user to re-run a command that already worked. Drop the second save, or comment what it is for. -
A failed restart can block for a long time before the error surfaces.
service_still_serving(main.rs:8460) →load_managed_service→refresh_managed_service_runtime_liveness, which performs a live endpoint probe. With several failed restarts this is serialised per service beforebail_on_failed_service_restartsever prints. Consider the PID check before the HTTP probe on this path, or a shorter timeout when all you need is "is it still up". -
The restore path writes the marker non-atomically.
ActivationSnapshot::restore(main.rs:8687) uses a barefs::write, while the forward path inwrite_active_runtime_markeruses temp-file + rename. The restore is the less likely path but the more costly one to get wrong — it runs when something has already failed. Reuse the same helper. -
Report rendering bypasses the shared component.
render_runtime_service_reconciliation(main.rs:8331) hand-rollswriteln!in thekey: valueshape thatapps/rocm/src/cli_report.rs::ActionReportexists to own.AGENTS.md§6 asks for the shared component to be extended rather than the pattern duplicated inline. The output is correct; this is drift. -
Dead parameter.
pin_service_record_to_runtime(main.rs:8530) takesruntime_key: Option<&str>and has aNonearm that setsrecord.runtime_id = None. Every call site passesSome(..). Take&str. -
README omits a step the error text includes.
README.md:415givesrocm services restart <service-id> --yesas the recovery; the actual bail message (main.rs:8603) says to checkrocm services logs <id>first. Worth matching, since a restart that just failed is likely to fail again unread. -
Rollback test is weaker than its activate twin.
runtime_rollback_reports_live_service_on_previous_runtime(main.rs:32104) asserts the formatted entry line but notservices_on_previous_runtime: 1; the activate version (main.rs:31927) asserts both. Removing the count line would leave the rollback test green.
Tradeoffs
-
Ambiguous family id classifies as stale (reviewer-focus b). Your reasoning holds:
runtime_manifest_for_selectorreturningNonemeans the record genuinely does not say which version, and stale is the conservative direction for the upgrade case. Worth being explicit that the cost is a needless stop/respawn under--restart-services, and that the mitigation isrefresh_from_engine_statenow preferringrequested_runtime_id— which shrinks the window only for records written after this change. -
update --apply --activatereports stale services but cannot act on them.--restart-servicesexists only onActivateandRollback, so that path names the servers and requires a second command. Reasonable scoping; worth one line in the PR body so it reads as a choice. -
--yeswithout--restart-servicesparses and does nothing. Already in your known residuals. A claprequires = "restart_services"is a two-line fix if you want it closed here.
Reviewer-focus answers
(a) Write-before-restart ordering. Correct, and the reasoning checks out. restart_internal_managed_service (main.rs:18013) calls load_managed_service first and builds serve_args from the record on disk (main.rs:18045), and builtin_engine_serve_http_args (main.rs:20889) gates --runtime-id behind if env_id.is_none() — so clearing env_id is what makes the pin reach the child. env_root_for_service (main.rs:4604) agrees. The failure branch restores both fields from the pre-write read, so the record never describes a runtime the server did not load. Device policy is re-validated: parse_device_policy at main.rs:18033 still bails on cpu_only, so no CPU fallback is introduced.
(c) Failed restart leaves the server stopped. Verified across all three surfaces — bail message (main.rs:8603), README, and the manual test's observable check. They agree with each other and with the code.
(d) See blocking #2.
Positive signals
ActivationSnapshotcaptures exactly the three fieldsactivate_runtimeandrollback_runtimemutate —default_runtime_id,active_runtime_key,previous_runtime_key— with no field mutated but uncaptured, and the capture genuinely precedes the first write on both paths.MarkerSnapshotdistinguishingAbsent/Unreadableis the right shape: restoringAbsentremoves the file it wrote, andUnreadablecorrectly does nothing rather than overwriting bytes it never read.- Temp files are created as siblings of their targets in both
RocmCliConfig::saveandwrite_active_runtime_marker, so the rename cannot cross a filesystem, and both clean up the temp on a failed rename. restore_in_memoryon a failedconfig.saveis a subtle one to have found — without it the caller's nextconfig.savewould re-persist the activation that was just refused.- The
--restart-servicesapproval gate is placed before every write on both handlers, so a refusal leaves the previous runtime fully in place. - Commit messages say what changed and why, all 14 carry DCO sign-off, and the two new
@id:tags follow the file's existing slug convention and are unique across the feature set.
Note on the PR description
Two claims in the body are now stale and contradict live CI on this head:
- "Real-hardware lanes have not reported... Nothing in this change has therefore been exercised on real hardware yet." All self-hosted GPU lanes now pass — MI300X, MI350P, rad3 R9700, Strix Halo Ubuntu and Windows. Only Strix Halo WSL2 is still pending.
- "The commits on this branch are unsigned, so the blocking
commit-signaturesCI gate fails and will keep failing."Commit signatures + sign-offpasses.
AGENTS.md §4 asks for live state before any external claim; worth refreshing both before this leaves draft.
The rocm install sdk behaviour in blocking #2 is also undescribed — the body's reviewer-focus (d) lists install sdk among the commands that can newly fail, without noting it is the one path that gains nothing in exchange.
🤖 Automated review by agent-hub — silo-review-and-fix on AMD AgentHub.
Model claude-opus-5[1m] · took 17m · diff caf925ea51cd · run
Not a human review. Reply here if a finding is wrong.
Round-2 fixes: all three blocking findings addressedEach blocking finding from the review above was re-verified against source before being fixed — both code findings confirmed, with one correction to the reasoning noted below.
1.
|
5eb9e80 to
ee18933
Compare
nowycondro
left a comment
There was a problem hiding this comment.
Summary
Change type and scope: bug fix + feature — makes rocm runtimes activate / rollback transactional across its two persisted writes, and adds live-service reconciliation with an opt-in --restart-services. Round 2, reviewing head ee18933 (18 commits, 8 files, +2320/-48) against merge-base 99e3bd5.
Overall assessment: Needs work. The transaction half (snapshot / restore / atomic writes) still holds up — I tried again to find a path that mutates a field the snapshot does not capture, or reaches a write before the capture, and could not. Of the three round-1 blocking findings, one is genuinely fixed, one is partially fixed, and the fix for the third introduces a concrete new failure: the new GPU scenario cannot pass on four of the six self-hosted lanes it targets.
What was and was not executed. There is no Rust toolchain in this sandbox (cargo: command not found). Nothing was compiled; no test, cargo fmt or cargo clippy was run here. Every code finding below is static analysis traced through source, and each blocking finding was re-verified by an independent adversarial pass. What was executed: live gh queries against the current head, the AGENTS.md §2 leak scan over origin/main..HEAD (clean — every internal/private hit is restart_internal_managed_service or "engine-private key"), a conflict-marker scan (clean), and the @id: duplicate check (grep -rho '@id:[a-z0-9-]*' … | sort | uniq -d → empty).
Live state, re-checked at the end of this review (AGENTS.md §4). The branch head moved during the review: 5eb9e80 → ee18933. git diff 5eb9e80..ee18933 is empty — the last four commits were re-signed, not rewritten, so everything below applies unchanged.
- Every hosted check on
ee18933passes, includingCommit signatures + sign-off— the commits are now GPG-verified (.commit.verification.verified == true). The PR body's signing section is out of date. E2E self-hostedforee18933(run35891325633) is queued and has not started a single job. The GPU lanes have still not reported for any head of this branch. That run is also the first thing that will hit blocking #1.
Round-1 findings status
Round-1 blocking #1 — env-pinned services are never moved: FIXED, with a half-fixed test
The None if record.env_id.is_some() => Matches arm is gone; classify_service_runtime_state now falls through to None => ServiceRuntimeState::Unknown (apps/rocm/src/main.rs:8288-8291). The doc comment (main.rs:8229-8253), README.md:400-402, docs/testing.md:333-335 and docs/manual-testing.md:261-263 now all say the same thing — env_id is not a pin. The code/user-text divergence round 1 flagged is genuinely closed, in one commit, across all four surfaces (AGENTS.md §5 honoured). Unknown is still reachable in production, so no dead arm was left behind.
Round 1 also asked for a test fixture resembling what vLLM actually writes. The new test writes {"status": "starting", "runtime_id": OLD_RUNTIME_KEY} (main.rs:32069-32075). A real vLLM launch writes both requested_runtime_id and runtime_id (engines/vllm/src/lib.rs:2015-2016), and refresh_from_engine_state prefers requested_runtime_id (crates/rocm-core/src/lib.rs:7740-7745). The fixture exercises only the fallback key, so if that precedence were reversed this test would not notice. It does now assert behaviour rather than the arm, which is the important half.
Round-1 blocking #2 — install sdk throws the reconciliation away: PARTIALLY FIXED
The field exists (main.rs:9399), is carried (main.rs:11259) and is rendered unconditionally ahead of the rollback hint (main.rs:11212-11216). On the common install path this is now correct and the round-1 objection is answered.
Two gaps remain: on the divergent-root path the reported count is computed against the wrong directory (non-blocking #1 below), and no test covers a non-empty carry (non-blocking #9), so the regression the fix exists to prevent would not be caught.
Round-1 blocking #3 — no automated coverage of the --restart-services --yes success path: NOT EFFECTIVELY FIXED
A scenario was added, but it cannot pass on most of the lanes it targets (blocking #1), the PR text still declares the gap (blocking #2), and two of its three assertions do not assert what their step names claim (non-blocking #10, #11).
Round-1 non-blocking findings — all seven untouched
| Round-1 item | Status at ee18933 |
|---|---|
config.save runs twice on the update path |
Untouched — main.rs:18440 still config.save(paths)?; right after activate_runtime at 18439, no mutation between |
| Failed restart blocks on a live probe before the error surfaces | Untouched — service_still_serving (main.rs:8471) → load_managed_service, serialised per service |
| Restore path writes the marker non-atomically | Untouched — ActivationSnapshot::restore still fs::write(&path, bytes) at main.rs:8699 |
Report rendering bypasses ActionReport |
Untouched — main.rs:8343 hand-rolls key: value; cli_report::ActionReport is already used at main.rs:3122, 7438, 7858, 8013, 9069 |
Dead Option<&str> parameter |
Untouched — pin_service_record_to_runtime (main.rs:8538) |
| README omits the log-check step the error names | Untouched — README.md:416 vs main.rs:8615 |
| Rollback test weaker than its activate twin | Untouched — main.rs:32258-32263 asserts only the entry line |
The three round-1 tradeoffs are unchanged; the clap definitions at main.rs:712-731 still carry no requires.
Blocking
1. The new GPU scenario cannot pass on the four self-hosted lanes where lemonade is the default serve engine
tests/e2e-cucumber/features/runtime_lifecycle.feature:105-112 (@id:runtime-lifecycle-activate-restart-services-succeeds)
The scenario's precondition is And a model is being served on GPU (feature:108), whose step (tests/e2e-cucumber/tests/e2e/serving_steps.rs:490) serves host_serve_target() (serving_steps.rs:411-419):
if e2e_cucumber::capability::host_capability().effective_serve_engine == "lemonade" {
("Qwen3-0.6B-GGUF", "lemonade", "Qwen3-0.6B")
} else {
("Qwen/Qwen3.5-0.8B", "vllm", "Qwen3.5-0.8B")
}effective_serve_engine (tests/e2e-cucumber/src/capability.rs:46-57) returns "lemonade" on native Windows and on any family that is not *-dcgpu / gfx906 / gfx908 / gfx90a. Of the lanes the PR body names — MI300X, MI350P, rad3 R9700, Strix Halo Ubuntu / Windows / WSL2 — only the two Instinct lanes resolve to vLLM. The other four serve lemonade.
The setup step And the running service is recorded on a different runtime does not rescue it: mark_services_on_other_runtime (runtime_lifecycle_steps.rs:628-674) rewrites runtime_id in the service record and in the engine state file, but leaves record.engine as "lemonade". And classify_service_runtime_state returns Matches on the engine check before it ever compares runtimes:
// apps/rocm/src/main.rs:8259
if engine_manages_own_runtime(&record.engine) {
return ServiceRuntimeState::Matches;(engine_manages_own_runtime is engine == "lemonade", main.rs:4483-4485.) So on those four lanes services.stale is empty → restart_stale_runtime_services (main.rs:8435) iterates nothing → services.restarted stays empty → render_runtime_service_reconciliation only emits the services_restarted: block when it is non-empty (main.rs:8375) → the Then step asserting out.contains(" services_restarted: 1") (runtime_lifecycle_steps.rs:555-558) fails.
The scenario is not skipped there: its only tags are @id: and @requires-gpu, and with no @requires-engine the engine gate falls back to the host default (expectation.rs:202-206, :500) — lemonade, which is available — so no skip. expectations.toml has no xfail row for it.
The repo's own docs state the premise this scenario violates, at docs/manual-testing.md:258-261: "A Lemonade server is never counted, because that engine brings its own runtime… Use a vLLM server on Linux or WSL to see a non-zero count."
The scenario's second assertion (services_on_previous_runtime: 0, steps.rs:564-567) would pass on those lanes for entirely the wrong reason, so it gives no compensating signal.
Suggested fix: give the scenario its own Given that serves with an explicit --engine vllm, and tag it @requires-engine:vllm so it skips cleanly where vLLM cannot start — or plant a vLLM-engine service record directly instead of reusing the shared serve step. Either way the queued E2E self-hosted run on ee18933 should be read before this leaves draft; it is the cheapest confirmation available.
2. AGENTS.md §3 is not discharged: the PR text still declares the gap and does not name the gated lane
§3: "if the scenario can only run on a gated lane (@requires-gpu, @nightly), say so in the PR text and name the lane that will exercise it."
The PR body's Known residuals still reads, verbatim:
No automated coverage of the
--restart-services --yessuccess path. … The success path is covered only by the manual test indocs/manual-testing.md.
Commit d97a1c1 contradicts it. The body also still says "The two new scenarios runtime-lifecycle-08 … and runtime-lifecycle-09" — there are now three. A maintainer reading the body would conclude the success path is uncovered and would not know to look for a GPU-lane result.
Suggested fix: replace that residual with a statement naming @id:runtime-lifecycle-activate-restart-services-succeeds, its @requires-gpu gate, the e2e-selfhosted lane — and, after blocking #1, which of those lanes can actually run it.
3. The comment justifying the env_id clear is false for an explicit --env-id launch, and the same false claim is in the PR body
apps/rocm/src/main.rs:8488-8494 (doc block on restart_service_onto_runtime), last sentence:
"The engine writes its own
env_idback into its state on the next launch, so nothing is lost by dropping it here."
PR body, reviewer focus (a): "Nothing is lost by dropping env_id: the engine writes its own back into its state on the next launch."
--env-id is a real user flag on rocm serve (main.rs:385-389), declared conflicts_with = "runtime_id" — the user is choosing an environment instead of a runtime. ManagedServiceRecord (crates/rocm-core/src/lib.rs:7575-7578) carries runtime_id and env_id and no field distinguishing a user-requested env from an engine-reported one.
Failure scenario: user runs rocm serve <model> --env-id my-env, then rocm runtimes activate <other> --restart-services --yes. pin_service_record_to_runtime (main.rs:8541-8551) sets runtime_id = Some(<other>) and env_id = None; builtin_engine_serve_http_args (main.rs:20909-20913) then emits --runtime-id <other> and omits --env-id. The server comes back in <other>'s environment, not my-env. What the engine writes back afterwards is <other>'s env, so the record never recovers the user's choice — and restore_service_record_pin (main.rs:8556-8565) only runs on failure.
To be fair to the design: given no field distinguishes the two cases, moving the service is a defensible choice, and the user can restore their intent by re-running rocm serve --env-id my-env — the environment itself is not destroyed. What is not defensible is a doc comment and a PR body asserting the opposite of what the code does about a destructive write on the success path. This is blocking as a contract-accuracy defect, not as a behaviour change.
Suggested fix: correct main.rs:8488-8494 and the PR body to say that an explicit --env-id pin is discarded and the server moved onto the activated runtime; add one sentence to README.md:406-408 warning --env-id users, since that is the one case where --restart-services does something they did not ask for.
Non-blocking
install sdkreports the reconciliation from the wrong root when the install root diverges from the caller's data root.main.rs:11234-11259:activation_paths = paths.clone().with_managed_root(manifest.install_root.clone(), false); whenpaths.data_dir != activation_paths.data_dirthe activation againstpaths— the user's real data root, where their running servers live — is discarded withlet _ =(main.rs:11245), and the reportedservicescomes from the activation againstactivation_paths(11253), whoseservices_dir()(data_dir/services,crates/rocm-core/src/lib.rs:1834) sits inside the SDK install tree and is normally empty. A user with a live vLLM server on runtime A, installing an SDK outside their managed data root, seesservices_on_previous_runtime: 0while their server has in fact been left behind. Two caveats that keep this off the blocking list: on the normal pip pathmanaged_runtime_data_root(crates/rocm-core/src/runtime.rs:518) strips back to the caller'sdata_dir, so the branch does not fire; and the comment atmain.rs:11243-11245says the discard is deliberate. (One correction to that condition:with_managed_rootnever touchesconfig_dir, so thepaths.config_dir != activation_paths.config_dirhalf of theifcan never be true.) Worth a decision rather than a silent 0 — capturing the11245result and using itsserviceswhen the branch fires is a two-line change.ActivationSnapshot::restorestill writes the marker non-atomically —main.rs:8699uses a barefs::writewhile the forward path (write_active_runtime_marker) andRocmCliConfig::saveboth use temp+rename. This runs when something has already failed; a truncated marker there leaves config and marker permanently disagreeing. (Round-1 carry-over.)render_runtime_service_reconciliationbypasses the shared component —main.rs:8343hand-rolls thekey: valueshape AGENTS.md §6 asks be taken fromapps/rocm/src/cli_report.rs::ActionReport, which this same file already uses five times. (Round-1 carry-over.)- Dead parameter —
pin_service_record_to_runtime(…, runtime_key: Option<&str>)(main.rs:8538); both call sites passSome(..), so theNonearm at8548is unreachable. Take&str. (Round-1 carry-over.) - Redundant
config.saveon the update path —main.rs:18440, immediately afteractivate_runtimeat18439with no mutation between;activate_runtimealready persisted viasave_activated_config(main.rs:8934). A failure there reports an error after a fully successful activation. (Round-1 carry-over.) - README recovery text omits the step the error names first —
README.md:416gives onlyrocm services restart <service-id> --yes; the bail message (main.rs:8615-8616) says to checkrocm services logs <id>first. (Round-1 carry-over.) - Rollback test still weaker than its activate twin —
main.rs:32258-32263asserts the entry line but notservices_on_previous_runtime: 1. Dropping the count line from the rollback render would leave it green. (Round-1 carry-over.) --yeswithout--restart-servicesstill parses and does nothing —main.rs:715-718and728-731carry norequires. (Round-1 carry-over / listed residual.)- The new
install sdktest cannot catch the regression it exists to prevent —render_sdk_install_success_includes_service_reconciliation_section(main.rs:34194) renders aRuntimeServiceReconciliation::default()built by the fixture atmain.rs:34174. It would pass unchanged iffinalize_successful_sdk_installhardcodedservices: RuntimeServiceReconciliation::default()— which is the shape of non-blocking #1. the model endpoint responds after the restartdoes not probe an endpoint —runtime_lifecycle_steps.rs:570-581runsrocm services listand string-matches the model name, discarding that command's rc. A service that restarted into a crash loop still has a record and still passes. Either rename the step or issue an HTTP request againstworld.endpoint.… names the service under services_restartedasserts the count, not the name —steps.rs:555-558checksservices_restarted: 1only, while the renderer emits- {service_id}right below (main.rs:8377-8379) and the sibling stale-service step (steps.rs:529-533) does assert the full entry line. (Theexits 0half is real —ok_outputassertsrc == 0.)rewrite_engine_state_runtimeis silently best-effort —steps.rs:678-700returns on any read/parse/write failure. If a state file cannot be rewritten,refresh_from_engine_staterestores the real runtime key, the service classifies asMatches, and the scenario asserts the wrong thing with no diagnostic pointing at the setup.- New README prose does not parse —
README.md:387: "Both commands count the servers that leaves behind and name each one". - README does not mention that
install sdknow prints the same summary —README.md:402-403names onlyrocm update --apply --activate, whileROLLBACK_RECOVERY_HINT's doc comment (main.rs:18456-18460) names all three call sites andrender_sdk_install_success(main.rs:11215) now emits it. --helpis inaccurate for this flag pair —main.rs:717and730document--yesas "Do not ask for interactive confirmation." There is no prompt on this path:ensure_service_restart_approved(main.rs:8410-8414) hard-errors. Neither flag's doc comment mentions the dependency, while README anddocs/testing.mdboth say "requires--yesand never prompts". AGENTS.md §5 lists--helpalongside README and docs as a surface to keep in sync.- Asymmetric rendering is undocumented —
services_on_previous_runtime:always prints (main.rs:8345-8349) whileservices_with_unrecorded_runtime:prints only when non-empty (main.rs:8359);README.md:397-399,docs/testing.md:331-333anddocs/manual-testing.md:264-265describe the two symmetrically, so a tester cannot tell "absence is normal" from "counter missing".
Tradeoffs
env_idsemantics are now a one-way door. Declaringenv_idnot a pin is the right call of the two round 1 offered — both engines write it unconditionally (engines/vllm/src/lib.rs:2015,engines/lemonade/src/lib.rs:3665) andrefresh_from_engine_stateadopts it, so the field cannot carry user intent. The cost is blocking #3. The alternative — arequested_env_idset only from the CLI flag — is a larger change; worth saying it was considered and deferred rather than leaving the choice implicit.- Ambiguous family id still classifies as stale (
main.rs:8274-8281) — unchanged from round 1, and the reasoning still holds. update --apply --activatereports stale services but cannot act on them —--restart-servicesexists only onActivate/Rollback;append_update_activate_summary(main.rs:18470) names the servers and the user needs a second command. Confirmedbail_on_failed_service_restartsis correctly not called on that path.
Positive signals
- Dropping the
env_idarm rather than widening it was the right of the two options, and the follow-through was complete across code, README,docs/testing.mdanddocs/manual-testing.mdin a single commit. runtime_activation_reports_service_with_no_recorded_runtime_as_unknown(main.rs:32104) is a genuine behavioural test: it pins that anUnknownservice is reported, is not counted inservices_on_previous_runtime, and is not touched byrestart_stale_runtime_services— three properties that fail independently.ee18933is a clean clippy fix: extractingrewrite_engine_state_runtime(steps.rs:674-700) flattens four nestedif lets intolet … elseguards with no behaviour change, and the commit message says exactly that.refresh_from_engine_statepreferringrequested_runtime_idover the resolved family form (crates/rocm-core/src/lib.rs:7731-7745) fixes the family-id ambiguity at the right layer, and degrades correctly for lemonade, which writes onlyruntime_id.- The
ActivationSnapshot/MarkerSnapshotdesign and therestore_in_memorysplit remain correct on a second adversarial pass.
Missing elements
- No test — unit or e2e — covers
install sdkorupdate --apply --activatecarrying a non-empty reconciliation through to rendered output. Both paths now print the section; neither is proven to print a true count. - No e2e scenario covers
rocm update --apply --activatereconciliation output at all.
Note on the PR description
The body has not been edited since f2984db; four commits have landed since, and the branch has been re-signed. Stale or now-false claims, quoted:
- "The commits on this branch are unsigned, so the blocking
commit-signaturesCI gate fails and will keep failing until someone with a registered signing key re-signs the range." — no longer true.Commit signatures + sign-offpasses onee18933and the head commit reportsverification.verified == true. Someone evidently did exactly that. - "all 10 commits" / "DCO
Signed-off-by:trailers are present on all 10 commits" — there are 18. - "head now
f2984db" — head isee18933. - "No automated coverage of the
--restart-services --yessuccess path." — contradicted byd97a1c1; see blocking #2. - "The two new scenarios
runtime-lifecycle-08… andruntime-lifecycle-09" — there are now three. - "Real-hardware lanes have not reported… Nothing in this change has therefore been exercised on real hardware yet." — still literally true for this head (run
35891325633is queued with no jobs started), but the reason has changed and round 1 recorded GPU lanes passing on an earlier head. Worth refreshing rather than leaving as-is (AGENTS.md §4). - The test-plan table (
cargo test -p rocm --bin rocm runtime→ 107 passed,cargo test -p rocm-core --lib→ 410 passed) was measured onf2984db; three tests have been added since, so the numbers no longer describe the tree. - "Nothing is lost by dropping
env_id: the engine writes its own back into its state on the next launch." — false for an explicit--env-idlaunch; see blocking #3.
Still accurate on re-check: expectations.toml has no rows referencing this work, and the @id: duplicate check is clean.
Open questions I resolved rather than asking (this review runs headless, with nobody to ask): I put the install sdk divergent-root count at the top of non-blocking rather than in blocking, because the branch is unreachable on the normal install path and the code comments the discard as deliberate — if that deployment shape is real for you, treat it as blocking. And I scoped blocking #3 to the comment and the PR-body claim rather than to the behaviour, since the behaviour is defensible given no field distinguishes a user env from an engine-reported one.
🤖 Automated review by agent-hub — silo-review-and-fix on AMD AgentHub.
Model claude-opus-5[1m] · took 48m · diff 1bb8a5f45dc7 · run
Not a human review. Reply here if a finding is wrong.
GPU lanes reported — one real finding in my own scenario, now fixedThe self-hosted lanes have now run It works. On MI300X and MI350P the scenario ran and passed every step, including It was wrong on Lemonade-default hosts. The scenario's premise is a server the activation can move, and only an engine whose runtime is a ROCm runtime can be moved. Lemonade brings its own, so
Fixed in This is exactly the failure mode flagged earlier: the scenario was written in a sandbox with no compiler and no GPU, so nothing local could have caught it. The GPU lanes did. The other two lane failures are not this change
Still needs a person
The PR description has been refreshed: the "Real-hardware lanes have not reported" and E2E-scenario paragraphs were stale and now describe what actually ran, and two residuals were rewritten — the restart-success coverage note, and a new one recording that 🤖 Fixes applied by agent-hub — |
05e85a6 to
1b88d17
Compare
1b88d17 to
e4a9989
Compare
nowycondro
left a comment
There was a problem hiding this comment.
Summary
Change type and scope: bug fix + feature — makes rocm runtimes activate / rollback transactional across its two persisted writes, and replaces a hardcoded closing note with live-service reconciliation plus an opt-in --restart-services. Round 3, reviewing head e4a9989 (1 commit, 9 files, +2374/−51) against merge-base 8788394.
Overall assessment: Needs work. The transaction half is careful and I could not break it — the Absent / Unreadable marker distinction, the in-memory restore on a failed first write, and the temp-file-plus-rename on both writes all hold up under the failure orderings I walked. The reconciliation half is correct in its classification but has one structural project-convention violation, one scenario-coverage gap, and a PR description that now contradicts the code on a decision round 1 specifically changed.
Three blocking items, all cheap. Nothing in the transaction logic blocks.
Blocking
1. A new subsystem with its own types was grown inside main.rs
apps/rocm/src/main.rs:8538-9007 adds ~470 lines of production code and seven new owned types — ServiceRuntimeState, RuntimeServiceEntry, FailedServiceRestart, RuntimeServiceReconciliation, ServiceRuntimePin, ActivationSnapshot, MarkerSnapshot — with no new module. main.rs is now 37,994 lines.
docs/architecture.md:15-17 makes this the deciding criterion, not a judgement call:
New subcommands and subsystems default to their own file from day one — they should not grow inside
main.rs/lib.rswaiting for a future extraction pass.
Full domain extraction — a subsystem's domain implementation moves into its own file that owns its own types (structs/enums), not just relocated functions; that ownership is what distinguishes this pattern from mechanical relocation […] This is the default for new subsystems.
The cluster owns seven types, so the mechanical-relocation escape hatch (automations.rs, uninstall.rs — "no owned types") does not apply. docs/architecture.md:30 notes main.rs is not yet modularized (EAI-7768), but that is precisely the pass this rule says new subsystems must not wait for. AGENTS.md §6 repeats it verbatim.
Fix: move the reconciliation and restart cluster to apps/rocm/src/runtime_services.rs as a private mod reached by qualified paths. RuntimesCommand and fn runtimes() stay in main.rs — that is the documented pattern for a subsystem with its own clap subcommand (docs/architecture.md:17 names RuntimesCommand/runtimes() explicitly). ActivationSnapshot / MarkerSnapshot are arguably dispatch-adjacent to activate_runtime and could stay; the seven-type reconciliation cluster is not.
2. The PR description contradicts the code on env_id
What changed, second bullet:
Two classes are deliberately never stale: engines that manage their own runtime […] and services pinned to an explicit
env_id(launched against that environment, not against whatever is active).
The code says the opposite, at length — apps/rocm/src/main.rs:8607-8621:
One class of live service is deliberately never stale: […]
env_idis deliberately NOT treated as a pin. Both vLLM and Lemonade write it unconditionally on every launch […] so treating it as a pin would silently exclude every service from reconciliation and defeat the feature entirely.
README.md:401, docs/testing.md:334 and docs/manual-testing.md:262 all state the correct behaviour. So do this PR's own Reviewer focus (a) and Known residuals. Only the "What changed" summary — the part a reviewer reads first — carries the pre-round-1 claim.
Fix: delete the env_id clause from that bullet; one class is never stale, not two.
While in there, the same section is stale about the branch: it describes 10 commits rebased to head f2984db, then 18 commits and a re-signing exercise mapping bf2556f→3996868 and so on. The branch is now a single commit, e4a9989, and none of those SHAs are reachable. The CI narrative under Test plan is stale in the PR's favour and should be re-stated: at this head, Strix Halo WSL2, Strix Halo Windows and rad3 R9700 are all green (the @requires-engine:vllm gate and the rebase did their job), MI300X is green, MI350P is still running, and the only failure is Strix Halo Ubuntu — GPU preflight (bounded wait for an available GPU), no test step reached, the same leaked-serve infrastructure problem the description already identifies.
3. rocm runtimes rollback --restart-services has no scenario
AGENTS.md §3 is unambiguous, and it is a project instruction:
If the change alters what a user of the CLI can observe — command output, exit codes, files or paths the CLI creates […] — then that behavior must be covered by a Gherkin scenario […] a unit test asserting the internal helper does NOT discharge this.
--restart-services on activate is covered twice (@id:runtime-lifecycle-activate-restart-services-requires-yes, and the GPU-gated …-succeeds). The same flag on rollback — apps/rocm/src/main.rs:8319-8328 — has neither a scenario nor a mention in the PR text. The only rollback assertion in the unit test is the non-restart case, and its message describes the wrong command (apps/rocm/src/main.rs:33343):
ensure_service_restart_approved(false, false, "rollback").is_ok(),
"an activation that restarts nothing needs no approval"Nothing anywhere asserts that the rollback refusal names rocm runtimes rollback --restart-services --yes rather than the activate spelling — which is exactly the kind of copy-paste error the hint exists to avoid.
Fix: one scenario mirroring runtime-lifecycle-09 against rollback, reusing the existing steps, plus an ensure_service_restart_approved(true, false, "rollback") assertion on the hint text. Both are ungated and run on the hosted mock lane.
Design review — the two questions you asked for a second opinion on
(b) Ambiguous family → Stale — the resolver cannot tell ambiguity from no-match
runtime_manifest_for_selector (apps/rocm/src/main.rs:4915-4933) returns None in two cases, and classify_service_runtime_state collapses both to Stale:
- no match at all — the recorded key names a runtime that is gone or was hand-edited.
Staleis the right answer. - ambiguity — two installs share the family id.
Staleis a guess.
For the report-only path the guess is advisory. For --restart-services it is not: restart_stale_runtime_services hands every stale entry to restart_service_onto_runtime, which stops the server (in-flight requests lost) and respawns it through a real readiness probe — a full model reload — for a server that was already on the target runtime. There is no late guard that notices the record already names the right key.
Your "shrinking legacy concern" argument holds up for the forward path: vLLM writes requested_runtime_id (engines/vllm/src/lib.rs:2485) and the refresh_from_engine_state change now prefers it, so new records carry the exact key; lemonade never reaches this code because engine_manages_own_runtime short-circuits first. But the conflation means the conservative direction is chosen for a case where it costs real downtime, and it is cheap to separate: give the resolver a tri-state (Exact / Ambiguous / NoMatch) and map Ambiguous to Unknown — reported under services_with_unrecorded_runtime, never restarted, with the same "restart it yourself" advice unknowns already get. NoMatch stays Stale, which preserves the upgrade case the feature exists for, because a single installed version is never ambiguous.
Non-blocking, but this is the one place where the classifier's error costs a user an outage rather than a wrong line of output.
(d) Reconciliation before the writes — the trade is right for two of the four commands
The failure surface is narrower than the description implies. load_managed_services (apps/rocm/src/main.rs:19482-19512) skips a corrupt record silently (if let Ok(mut record) = serde_json::from_slice(…)) and returns empty for a missing directory. Only two things are actually fatal: read_dir on the services directory failing, and fs::read on one .json returning an error. So "unreadable services folder" means permissions or a broken mount, not a malformed record.
The latency is the part worth more attention than the description gives it. Per live service in ready/running: a 750 ms listing check (SERVICE_LIVENESS_CHECK_TIMEOUT, apps/rocm/src/main.rs:19353) plus an 8 s inference probe (INFERENCE_PROBE_TIMEOUT, crates/rocm-core/src/lib.rs:685), throttled per-record by a 15 s retry interval (…:705) but sequential across services with no aggregate ceiling. Three cold services is ~26 s in front of a command that was previously two file writes.
The codebase's own precedent splits cleanly, and not where this change lands:
| call site | on unreadable service state | |
|---|---|---|
render_services_text |
propagates Err |
main.rs:18009 |
build_service_prune_plan |
propagates Err |
main.rs:8007 |
render_internal_status_text |
propagates Err |
main.rs:18202 |
busy_gpu_indices |
degrades to empty | main.rs:21735 |
| uninstall planner | .unwrap_or_default() |
main.rs:20688 |
existing_live_managed_service |
.ok()? |
main.rs:19334 |
Commands whose purpose is to report service state propagate. Call sites where service state is ancillary to a different primary action degrade.
By that line, rocm runtimes activate and rollback should stay fatal — the user asked for exactly this, the command has written nothing yet, and retry is free. But finalize_successful_sdk_install and apply_runtime_update are in the ancillary bucket, and the asymmetry is real: by the time activate_runtime is reached in finalize_successful_sdk_install (main.rs:11610-11648), a multi-gigabyte download and install has completed and setup.completed is already saved — so a chmod-broken services directory turns a successful 20-minute install into a non-zero exit with the SDK on disk but not active, and the obvious recovery (rocm runtimes activate <key>) fails the same way.
Recommendation: keep the ? for the standalone commands, and give the install and update paths a variant that degrades to an empty reconciliation plus a named warning in the report. ~20 lines. Answering your framing directly: "fail clean rather than half-apply" is the right instinct, but it is only the user's instinct for the command they actually invoked.
Non-blocking
ActivationSnapshot::restorewrites the marker with a barefs::write(apps/rocm/src/main.rs:9085) while every other write this PR touches goes through temp-plus-rename. Safe today —restoreonly runs when the rename failed, so the destination still holds the original bytes and they are rewritten identically. But it is the one place the PR's own stated invariant ("a reader sees either the old config or the new one, never a half-written file") is not upheld, and the test atmain.rs:33766already callsrestoreafter both forward writes succeeded, which is the shape that would make it bite. Factor the temp-plus-rename into a sharedwrite_atomic(path, bytes)and use it in all three places.- A failed
fs::writeto the temp file strands it.crates/rocm-core/src/lib.rs:6180andapps/rocm/src/main.rs:12202both clean up on a failedrenameviainspect_err, but thefs::writeabove it uses a bare?, so a mid-write failure (full disk) leaves a partial.json.tmp-<ts>-<pid>beside the real file — in the marker's case, in the folderrocm runtimesreads. Add the sameinspect_errto the write. requested_runtime_idis not in the engine-protocol contract. The CLI now depends on it (crates/rocm-core/src/lib.rs:7753) butcrates/rocm-engine-protocoldoes not define it; vLLM writes it, lemonade does not (harmless today only becauseengine_manages_own_runtimeexempts lemonade before the field is read). AGENTS.md §6 calls that crate a contract surface and requires verifying all impacted engines after protocol changes. A third engine author has no signal to write the field, and would silently reintroduce the family-id degradation this PR fixes. Declare it, or document it where engines are told what to write into their state file.- The
refresh_from_engine_statechange is missing from the description. It is a behaviour change in a shared crate that alters whatrecord.runtime_idmeans after every refresh, workspace-wide — not only on the activation path. I traced every reader (main.rs:8638,:8835,:8932,:8949,:18462,:18471,apps/rocmd/src/lib.rs:5058) and none breaks: each either wants the exact key or passes it to an engine that accepts both forms, and pre-existing state files lack the field so the fallback preserves today's value on upgrade. It is safe — it just isn't mentioned in What changed, which lists onlyRocmCliConfig::saveforrocm-core. - No integration-level test for the new preference. The record planted at
apps/rocm/src/main.rs:33192writes onlyruntime_id, so it exercises the pre-existing fallback. The prefer-requested_runtime_idpath is covered only by therocm-coreunit test. A variant planting a realistic vLLM state file —runtime_idfamily form andrequested_runtime_idexact key — and assertingclassify_service_runtime_stateresolves it would close the loop between the two halves of this change. - The double-fault arm of
restore_after_failed_activationis untested.main.rs:9135-9141has two arms with materially different messages; only the success arm is exercised. Makerestorefail (e.g. replace the config path with a directory after capture) and assert the "may now disagree" text and the re-run instruction. - README grammar.
README.md:386: "Both commands count the servers that leaves behind and name each one" — missing subject. "Both commands count the servers they leave behind". - Ctrl-C mid-loop leaves a partial move with no report.
restart_stale_runtime_serviceshas no signal handling, and the report is rendered only after it returns — so interrupting a multi-server move prints nothing at all and the user must reconstruct state fromrocm services list. Worth a line in the docs at minimum. --yeswithout--restart-servicesis already in your residuals;#[arg(long, requires = "restart_services")]is the one-line version. For the record, I checked the claim that the runtime is switched before the refusal and it is not true: the gate runs beforeactivate_runtimeon both paths (main.rs:8285,:8323), so declining does leave the previous runtime in place, exactly as the comment above it says.- AI-generated footer in the PR body. AGENTS.md §11 says "avoid AI-generated boilerplate footers"; the trailing block also carries an
agenthub.amd.comrun link, which is an organization-internal URL on a public repository and not openable by anyone outside. I have not edited the body: AGENTS.md §1 requires explicit approval before editing an upstream PR, and this run is review-only.
Checked and cleared
Raised during this review and refuted against source, so they are not re-litigated later: the // three of the four reports… comment at main.rs:8777 is correct as written — it counts the two flagless commands plus rollback, where the flag exists but points the wrong way, against activate as the fourth. And engine_manages_own_runtime keys off record.engine, not a pattern match on the runtime key string, so the lemonade exemption cannot misfire on a key that happens to look engine-private.
Positive signals
- The
MarkerSnapshot::AbsentvsUnreadablesplit is the detail most implementations of this get wrong: a read fault never licenses the restore to delete bytes it never held, andmain.rs:32973tests exactly that. restore_service_record_pinre-reads the record from disk rather than writing back a stale in-memory copy, so it cannot clobber the pidrestart_internal_managed_servicejust wrote — and it restoresenv_idas well asruntime_id.- Resolving a recorded family id through the same selector path every other runtime lookup uses, rather than string-comparing, is the right call and is what stops
--restart-servicesrespawning correctly-placed servers in the common single-install case. - The docs agree with each other and with the code on the
env_idpoint across all three surfaces — it is only the PR body that drifted. - Test isolation is clean:
test_pathscomposes pid and millisecond timestamp, noset_var, no fixed paths, and the three new@id:tags are unique across the feature directory.
Verification
No Rust toolchain in this sandbox — cargo is not installed — so I did not run fmt, clippy or any test suite, and I am not asserting the described local results either way. CI at e4a9989 is the evidence: 25 checks pass including clippy, build-and-test, windows-build-and-test, Test (affected crates), the hosted E2E tests lane, Commit signatures + sign-off and Sphinx docs build (-W); MI350P is still running; the single failure is Strix Halo Ubuntu at GPU preflight (bounded wait for an available GPU), which ran no test step. The AGENTS.md §2 leak scan over the diff is clean — the two internal hits are engine-private in prose and the two private hits are the same phrase. No conflict markers.
🤖 by agent-hub on AMD AgentHub
siloteemu
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · bc30f4d
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change (prw-base...HEAD, 4 commits, 13 files).
Title names EAI-7404, a private-tracker ticket that could not be retrieved from here. The description says the change makes runtime activation and rollback transactional and adds --restart-services / --yes.
Each finding is marked new (first raised this round) or standing (carried from the change request filed at 6bd837ba).
Blocking
-
[new] The branch no longer merges, and it re-implements a fix that is already on the base. —
apps/rocm/src/main.rs:9045-9075,tests/e2e-cucumber/features/runtime_lifecycle.feature(scenarios 08 and 13),tests/e2e-cucumber/tests/e2e/runtime_lifecycle_steps.rs,apps/rocm/src/main.rs(plant_service_record, ~28175)
prw-baseis 35 commits ahead of the merge base.git merge-treeconflicts inmain.rs,runtime_lifecycle.featureandruntime_lifecycle_steps.rs. Base already keepsprevious_runtime_keywhen the active runtime is re-selected, and carries scenario@id:runtime-lifecycle-reactivating-keeps-rollback-targetasruntime-lifecycle-08. This PR adds a near-copy as scenario 13 and reuses number 08 for a different scenario. Base also splitplant_service_recordinto a wrapper overplant_service_record_without_its_key. Taking this PR's version of that helper would drop that split.
Confidence 85 · architectural · Fix: rebase, keep base's reactivation branch and scenario, add only the newservices/snapshot parts, renumber the scenarios (this makes theruntime-lifecycle-10reference atruntime_services.rs:385stale), and buildplant_service_record_on_runtimeonplant_service_record_without_its_key. -
[new] A failing check on this head:
E2E tests (Strix Halo, Ubuntu). — CI run forbc30f4d4
The log shows 152 of 160 scenarios passed and 8 failed. One failure is unexpected:engine-shell-marks-the-prompt(the engine-shell step panicked with "the marker never reached a prompt line"). The diff does not touch that area. All of the newruntime-lifecycle-*scenarios that ran on that lane passed. This looks unrelated to the change, but the check is red and no passing rerun is visible. The same workflow also failed on a recentmainrun.
Confidence 70 · mechanical · Fix: rerun the lane after the rebase, or escalate with evidence if it still fails. -
[standing, re-found blind] The fix for "the restart comes back on the old runtime" can be reverted with every unit test still green. —
apps/rocm/src/main.rs:18348(apply_restart_runtime_pin(&mut record, runtime_pin)),apps/rocm/src/runtime_services.rs:414(Some(runtime_key))
Two reviewers ran this independently. Removing the call at 18348, or passingNoneat 414, leaves the fullrocmunit suite passing (980 and 986 tests).pinning_the_record_is_what_puts_the_new_runtime_in_the_restart_argvcalls the helper directly, and its own doc says it cannot catch a missing call. Scenarioruntime-lifecycle-10is GPU-only, and its comment says it cannot tell the orders apart with a single install of the family. The pure steps before the spawn (load record, apply pin, build argv) are testable without an engine.
Confidence 100 · logic · Fix: pull the "load, pin, build argv" part ofrestart_internal_managed_serviceinto a function the restart itself calls, and point the argv test at it. -
[new] A test claims an ordering check it can never fail. —
apps/rocm/src/main.rs:36362-36387(render_sdk_install_success_includes_service_reconciliation_section)
The fixture hasprevious_runtime_key: None.render_sdk_install_successprints the rollback hint only when that isSome. Soif let Some(rollback_pos) = rendered.find("rocm runtimes rollback")never matches, and the comment "must precede the rollback hint" is never checked. Moving the reconciliation after the hint left the test green (mutation run).
Confidence 100 · mechanical · Fix: useprevious_runtime_key: Some(..)and assert the hint position unconditionally. -
[new] Behaviour this change introduces is exercised by no test. — each confirmed by a one-branch mutation that left the suite green
main.rs:11442:services: activation.servicesininstall sdkfinalization. Replacing it with the default value passed everything.main.rs:8877: the message printed when restoring after a failed activation also fails. The comment atmain.rs:34679shows the case is reachable, then skips it because the text is "short enough to read correctly by inspection".main.rs:9065-9073: when the rollback target has been uninstalled, the marker should record no id for it. Taking the id fromcurrentinstead passed all tests.crates/rocm-core/src/lib.rs:7828-7831: an empty or blankrequested_runtime_idmust fall through toruntime_id. Swapping the filter for.next()passed all 5 scoped tests.main.rs:752and:765:--yeswithout--restart-servicesis refused by claprequireson both subcommands. Notry_parse_fromtest and no scenario covers it.
Confidence 95 · mechanical · Fix: add the tests named above. A unit test does not discharge the user-observable refusal under AGENTS.md §3, so add a scenario for it.
-
[new] User-visible failure output of a failed
--restart-serviceshas no scenario. —apps/rocm/src/runtime_services.rs:300-333,:510-556;runtime_lifecycle.featurescenario 10 covers only success and only on the GPU lane
Non-zero exit,services_restart_failed: N, the "still serving / stopped by the attempt" text and the record being put back are unit-tested only. A mock-lane scenario looks feasible.restart_internal_managed_servicerefuses a public-bind service without an endpoint key before it stops anything, so a planted live record with a public host would exercise the failure path with no GPU.
Confidence 70 · architectural · Fix: add the scenario, or say in the PR text which lane covers it. -
[new] Re-activating the active runtime prints a false
changed_from_runtime_key. —main.rs:9058-9072, printed atmain.rs:8468-8471
On the reactivation pathprevious_runtime_keyis now the retained rollback target, andruntimes()prints it aschanged_from_runtime_key: <old>although nothing changed. Before this change it read<unset>. The report's ownnote:tells users to run exactly this reactivation. Neither the unit test nor scenario 13 checks that line.
Confidence 75 · logic · Fix: carry a "reactivated" flag inRuntimeActivationResultand print<unchanged>, with the rollback target as a separate line. -
[new] A restart whose server never becomes ready is still counted as restarted, and the command exits 0. —
main.rs:18460-18473(existing code),runtime_services.rs:314-315and:414
When the readiness probe times out,restart_internal_managed_servicesets statusstartingand returnsOk. The restart is then listed underservices_restarted, the pin is not restored, andbail_on_failed_service_restartsdoes not fire. The weakness was there before. This change enlarges its blast radius by restarting a batch of servers onto a different runtime, where late failures such as ABI or out-of-memory are most likely, and reporting that as success. README.md:428-429 ("comes back serving from it") overclaims for the same reason.
Confidence 70 · logic · Fix: count a server as restarted only when readiness isServing, report anything else apart, and reword the README. -
[new] Dead code: the exact-match arm in
classify_service_runtime_state. —apps/rocm/src/runtime_services.rs:123;main.rs:38476
runtime_manifest_for_selectoralready matchesruntime_keyfirst, so the next arm covers the exact case and this arm cannot change the outcome. Disabling it (if false && ...) left all tests green. The test's doc claims "breaking it would stop and respawn servers for nothing", which is false.
Confidence 85 · logic · Fix: delete the arm, or keep it as a documented fast path and correct the test comment. -
[standing since
6bd837ba] The restart-order fix is not guarded by any test that runs in default CI — partly holds. —tests/e2e-cucumber/tests/e2e/runtime_lifecycle_steps.rs(stepre-activating that runtime reports the service already on it),runtime_lifecycle.feature,apps/rocm/src/runtime_services.rs,apps/rocm/src/main.rs
The blind pass did not re-find this objection in its original wording. After the blind pass, it was put to the reviewer together with the author's reply that it was closed in905c9c28, so this verdict was reached with the objection in view. The reviewer's verdict: the comment part is resolved, because the step and feature comments no longer present scenario 10 as verifying the order. The argv unit test that was asked for exists, but it exercisesapply_restart_runtime_pindirectly. The restart's call sites are unguarded, as the blocking finding above shows. A restarted server coming back on the old runtime can therefore still return with every lane green. This objection and the re-found blocking finding above have one root cause, and the fix above resolves both.
Non-blocking
-
[new] Comments that are wrong, or that describe the PR's own revision history.
main.rs:34685: namesruntime-lifecycle-11, but the directory-at-marker trick is scenario 14.main.rs:12019-12029and:33537-33542: say Windows "falls back toReplaceFileW".publish_temp_file(therock.rs:4735-4749) triesReplaceFileWfirst when the destination exists. The new rocm-core doc comment states it correctly, so the two disagree. The code governs.runtime_lifecycle_steps.rs:252-256: sayscreate_newagainst a directory fails. The temp file is created fine; the publish onto the directory fails.runtime_lifecycle_steps.rs:883-911: names<data>/services/<id>.state.json, but the engine state lives at<data>/engines/<engine>/state/<id>.json, so the.state.filter is dead.runtime_services.rs:99-100: citesengines/*/src/lib.rs, but the code is in thestate.rsfiles.main.rs:34586-34591: calls the test Unix-only "the same limitation" as a rocm-core twin that now has a Windows version, and refers to "the residual".crates/rocm-core/src/lib.rs:7819-7827and about:13040: "exact runtime key". The field holds whatever selector the launch requested, which may be a family id. Behaviour is fine; the wording is not.- History or PR references that go stale on merge:
main.rs:33788-33791("the oldNone if record.env_id.is_some()arm", which is not on base),lib.rs:12940-12946,:12985-12991,:6211-6213("follow-up on the PR that introduced this note", with no issue),runtime_lifecycle.feature:153-156and:178,runtime_services.rs:376-379.
Confidence 80 · mechanical · Fix: state the invariant, drop the history, point to an issue where one is meant.
-
[new] The manual-testing walkthrough probably cannot be followed as written. —
docs/manual-testing.md:242-296
It says section 2 "installed a second runtime", but wheel keys do not include--prefix, so section 2 normally replaces the section 1 runtime and no other runtime key exists. It also never installs vLLM into either runtime, and vLLM is per-runtime (env_root_for_runtime), so the serve and the move would fail early. These were traced from the code, not run.
Confidence 70 · logic · Fix: add a step that installs a different runtime androcm engines install vllm --runtime-id <key> --yesfor both. -
[new] README sample runtime keys have a shape the CLI never produces. —
README.md:411-413(release-wheel-gfx942-7-12-0; real keys arerelease-wheel-multi-arch-<ver>-<hash>,therock.rs:6177-6181)
Confidence 75 · mechanical · Fix: use placeholders. -
[new] Two
Thensteps run the CLI themselves. —runtime_lifecycle_steps.rs:778-790and:792-841(feature lines 176 and 135). They overwriteworld.cli_*from inside a#[then].
Confidence 70 · mechanical · Fix: make each aWhenfollowed by a plainThen. -
[new] A step asserts less than its wording says. —
runtime_lifecycle_steps.rs:658-667. "names the service under services_restarted" checks onlyservices_restarted: 1, not the id line. The next step's wording is garbled.
Confidence 85 · mechanical · Fix: assert the id, and reword the step. -
[new] Pre-existing flaky test in a touched hunk. —
main.rs:30041and:30132. Both usetest_paths("restart-public-no-key"). The root isname-pid-millis, so two tests starting in the same millisecond delete each other's directory. It failed in about 1 of 6 full runs. Not caused by this change.
Confidence 85 · mechanical · Fix: give each test a unique name.
Decisions for the author
-
Services are read before any write, so an unreadable services folder now refuses activation, rollback,
install sdkfinalization andupdate --apply --activate— tradeoff
Fail-clean means the previous runtime stays fully in place; the cost is a new failure point on paths that never read service state before, and on install and update it comes after the install itself succeeded. An I/O error on one record (for example one removed byservices prunebetween listing and reading) also fails it. The description discloses this as (d). Consider treatingNotFoundon an individual record as skip, and saying "installed but not activated" on the install and update paths. -
Win32_Storage_FileSystemis a production dependency ofrocm-core, used only by a#[cfg(windows)]test — non-blocking-improvement
apps/rocm/Cargo.tomlalready enables the feature, so there is no practical cost. A dev-dependency would say what it is for. -
RocmCliConfig::saveis atomic by rename but not fsynced, and uses plainfs::rename— tradeoff
This matches the old no-fsync behaviour. The doc comment admits that on Windows,config.jsonheld open without delete sharing makes a save fail with a sharing violation, whereas the marker write falls back toReplaceFileW. The ~60-line doc comment onsave(lib.rs:6171-6228) is longer than the guarantee it states.
Positive signals
- Services are read before either write, and an in-memory restore runs when
config.savefails. MarkerSnapshot::Unreadablestops the restore from deleting a marker whose bytes were never captured.- Approval is checked before anything is written, and
--yesis tied to--restart-servicesby claprequires. - What counts as "still serving" after a failed restart is read back from the record, not inferred from how far the attempt got.
- Mutation runs confirmed that scenarios 11, 12, 13, 14 and 15 each fail when their production branch is broken, and that the
service_restart_audit_outcomeseverity tests fail whenwarnis collapsed back toinfo. RocmCliConfig::saveno longer removes the destination before the rename (the old marker writer did), and the temp name includes the pid.
Deployment notes
- New flags
--restart-servicesand--yesonrocm runtimes activateandrollback. Without--restart-services,--yesis refused. - The activation report now prints
services_on_previous_runtime: Nand related lines instead of the fixed sentence, which is a visible output change for scripts that matched it. rocm runtimes activate,rollback,install sdkfinalization andupdate --apply --activatenow read the services folder, and an unreadable one makes them fail.config.jsonis written by temp file and rename, not fsynced.- The branch needs a rebase onto current
mainbefore it can merge.
What this covered
Every file in the change was read completely by seven independent workers. The workers were grouped by coupling: rocm-core and the atomic-write helper; the production activation and service code in main.rs and runtime_services.rs; the unit tests in main.rs; the Gherkin scenarios and step definitions; the docs; guidance and comments; and history. I checked the merge conflict, the scenario id already on base, and the CI summary myself. I also read the vacuous ordering test, the reactivation branch and the restart-pin call site. Range: 4ee61616… (prw-base) to bc30f4d4….
Runs: workers ran unit tests and one-branch mutations (about 50 in total) in private copies. They ran scenarios 08, 09 and 11-15 of runtime-lifecycle against the built binary on HEAD. Clippy (-D warnings) passed on HEAD. The rocm unit suite passed at baseline apart from the flaky test noted above, plus two build_metadata failures from a commit-signing setting in the review environment that is unrelated to this change.
Did not run:
#[cfg(windows)]code was not compiled or run here (no target); the Windows FFI was reviewed by reading only. The CIwindows-build-and-testcheck passed.- Scenario
runtime-lifecycle-10is GPU-gated and was not run. cargo build -p rocmfor running help output was abandoned in the docs pass, so documented flags were checked against the clap definitions instead.- The "passes with the production change reverted?" verdicts are by reasoning where a mutation could not be run. The base was not built.
- Not run:
ai-instruction-quality, because no agent-instruction file is in the diff. - Not retrieved: the EAI-7404 ticket (private tracker).
- Findings below the 70 confidence cutoff were dropped (resolver inconsistency with case-twin keys, per-service network probes during activation, the
install sdkreport root when the managed root moves, and others).
No prompt-injection content was found in the checkout. The findings were reached without the review discussion. Only the standing item above was weighed against the author's reply, and that happened after the blind pass.
Activation wrote the config and the active-runtime marker as two independent steps and told the user nothing true about the servers already running. A marker write that failed left the config naming the new runtime while the marker still named the old one — and the marker is what runtime resolution and the storage retention holds both read. The report closed with a fixed note, "running services keep their recorded runtime until they are restarted", printed whether or not a single server existed, so it could neither name a server really left behind nor stay quiet when there was none. Make the switch transactional and make the report derived state: - Capture the config fields and the marker bytes before the first write, and put both back when either write fails. "No marker" and "a marker this process could not read" stay distinct, so a read fault never licenses the restore to delete bytes it never held. - Write `config.json` and the marker through a temp file and a rename instead of truncating or unlinking the destination first, and clean the temp file up when the rename fails. The temp name carries the pid as well as the timestamp so two `rocm` processes cannot clobber each other. - Read the live service records before anything is written and report them: `services_on_previous_runtime: <n>` with a line per server, a separate `services_with_unrecorded_runtime:` bucket, and the count printed even at zero so "looked and found nothing" is distinguishable from "never looked". A recorded family runtime_id is resolved the way every other selector is, so a server already on the target runtime is not named as left behind. - Add `--restart-services` to `runtimes activate` and `runtimes rollback`, gated on `--yes` and checked before anything is written. Each server is re-pinned onto the new runtime and then restarted — in that order, because the restart rebuilds argv from the record — and its `env_id` is cleared in the same write, or the engine resolves a runtime nobody chose. A failure puts the record back, is named in the report, and exits non-zero; whether the server is still up is read back rather than inferred, because a restart refused before the stop leaves it serving. - Prefer the engine's exact `requested_runtime_id` over the resolved family id when refreshing a record, so a restart re-pins the version the service is really on. - Put the reconciliation subsystem in `apps/rocm/src/runtime_services.rs` rather than growing it inside `main.rs`. It owns five types, which is the criterion `docs/architecture.md` and AGENTS.md §6 use to require full domain extraction from day one. `RuntimesCommand` and `fn runtimes()` stay behind, which that doc names as the exception. - Leave `previous_runtime_key` alone when the requested runtime is already active. The report tells the user to re-activate the active runtime to move the servers without moving the runtime, and that note prints one line above "if this causes problems, run `rocm runtimes rollback`" — so deriving the previous key from the current manifest and writing it unconditionally meant following the first instruction destroyed the second. A switch that does not switch has no history to record and none to discard. - Drop the redundant `config.save` on the `update --apply --activate` path: `activate_runtime` already persisted the same struct, so the second write could only fail after an activation that fully succeeded. Restore the marker through the shared atomic-write helper rather than a bare `fs::write` — it runs when something has already failed and can least afford a torn marker. - Print the same reconciliation from `install sdk` and `update --apply --activate`, which activate a runtime too. Those commands take no `--restart-services`, and re-running `rollback` with it would switch the runtime a second time, so the note names the one invocation that moves the servers without moving the runtime: `rocm runtimes activate <runtime_key> --restart-services --yes`. Covered by unit tests for each branch of the snapshot restore, the classification, the failed-restart bookkeeping and the audit severity, and by `runtime_lifecycle` scenarios for the derived report, the refusal without `--yes` on both commands, the re-activation that keeps the rollback target, the unrecorded-runtime bucket, the two refusals that leave the previous runtime fully in place — an unreadable services folder, and a marker write that fails after the config write succeeded, each asserting the config and the marker rather than one of them — and, on the vLLM GPU lanes, a real server stopped, respawned and serving again from the runtime just activated, read back through the engine state so a respawn onto the old runtime cannot pass as success. Signed-off-by: Nowy <740902+nowycondro@users.noreply.github.com>
`rocm runtimes activate --restart-services --yes` wrote the new runtime key into the service record and then restarted the server. The write never reached the child. `restart_internal_managed_service` loads the record through `load_managed_service`, whose `refresh_from_engine_state` re-adopts the `runtime_id` and `env_id` of the process being replaced — still running at that moment, its state file still describing it. The pin written a line earlier was undone in memory before the argv was built, and the revived `env_id` alone was enough to drop `--runtime-id`, because `builtin_engine_serve_http_args` emits it only when `env_id` is `None`. The child came back up on whatever the engine resolved for itself while the report counted the server as moved. The MI300X lane caught this at 6bd837b: `runtime-lifecycle-10`'s closing step failed with `services_on_previous_runtime: 1`, naming the just-restarted server with `recorded_runtime=therock-release:gfx94X-dcgpu` — the family form an engine records when it was handed no `--runtime-id`. - Pass the runtime key to `restart_internal_managed_service` and re-apply it after the load, through `apply_restart_runtime_pin`. The write stays: the report and a restart that fails before the stop are both read from the record on disk, and it is what captures the pin to restore. `rocm services restart` and the sandbox restart tool pass `None` and keep the behaviour where the record follows the running process. - Assert the pin where it is observable without an engine. `pinning_the_record_is_what_puts_the_new_runtime_in_the_restart_argv` builds the child's argv from a record loaded over a live-shaped engine state file, with and without the re-application: without it the argv carries no `--runtime-id` at all, with it the argv names the activated key and no `--env-id`. Emptying `apply_restart_runtime_pin` fails it. This is the unit test the third review round asked for, and it closes that round's untested `env_id`-argv claim in the same assertions. - Cover `classify_service_runtime_state`'s family-resolution arm directly: one install carrying a family id resolves to it, and the same id is still stale against a runtime it does not resolve to. Only the ambiguous case was tested. - Stop `runtime-lifecycle-10`'s comments claiming its closing step gates the pin-before-restart order. It cannot: a single install of a family resolves the family form back to the activated runtime, so both orders report zero there. Say what the step does catch — a respawn on a runtime that does not resolve to the activated one, or one that records no runtime at all. Signed-off-by: Nowy <740902+nowycondro@users.noreply.github.com>
Section 4 of `docs/manual-testing.md` is where the PR text points a maintainer
with real hardware, and as written every new step in it ran against zero
eligible servers. Nothing earlier in the document leaves a server up — section
3 serves Lemonade and stops it on both of its branches, and section 4's own
preceding block stops whatever remained — so "With one up, check what an
activation says about it" stated a precondition nothing established. Re-reading
it as an instruction to restart section 3's server does not rescue it either:
Lemonade brings its own runtime and is deliberately never counted, so the
report would say `services_on_previous_runtime: 0`, print no `note:` line, and
the next instruction ("move it with the command the report's own note gives
you") would have nothing to read a command from. The whole walkthrough passed
vacuously, and the sentence asserting the server "is now named under
`services_on_previous_runtime`" was false for anyone following the steps.
- Add an explicit step that starts a vLLM managed server on the active runtime
and leaves it running, with the service id read back from `rocm services`
before going on, and say there why section 3's Lemonade server cannot stand
in for it. Drop the passive "With one up" framing.
- The expected-results bullet that carried this as an aside ("Use a vLLM server
on Linux or WSL to see a non-zero count") now points at that step instead of
standing in for it.
Also from the same review round:
- Give `RocmCliConfig::save`'s replace-not-rewrite guarantee a Windows test.
The only assertion that could tell a rename from an `fs::write` was
`#[cfg(unix)]` and read the inode; its two cross-platform siblings pass
against either implementation, so a Windows-only regression would have passed
every gate. The counterpart reads the same identity through
`volume_serial_number`/`file_index`, and the Unix test now names it so the
gate is not read as platform-neutral coverage.
- Separate the two merged doc blocks in `runtime_lifecycle_steps.rs`: the
marker-reading rationale had come to rest above the config-reading helper,
leaving the marker helper undocumented.
Signed-off-by: Nowy <740902+nowycondro@users.noreply.github.com>
The `#[cfg(windows)]` counterpart added last round for `RocmCliConfig::save`'s replace-not-rewrite guarantee could not compile. It read the file identity through `std::os::windows::fs::MetadataExt`, whose `volume_serial_number` and `file_index` accessors are still unstable behind the `windows_by_handle` feature gate, so the Windows lane failed with four `error[E0658]`s while every other lane stayed green -- nothing else in the workspace builds that arm, and no local gate here could have caught it either, since `cargo` only ever ran on Linux. - Read the identity through `GetFileInformationByHandle` instead, the stable Win32 call those accessors wrap, following the `#[allow(unsafe_code)]` FFI pattern this crate already uses. `rocm-core` gains the `Win32_Storage_FileSystem` windows-sys feature it needs; the other five stay as they were. - The identity is no longer optional. The `MetadataExt` accessors return `Option`, so the old test skipped its assertion when either was `None` and could not distinguish "Windows declined to report an identity" from "this test asserted nothing". `GetFileInformationByHandle` either fills both fields or fails, and a failure is now a test error. - Correct the Unix test's pointer to its counterpart: it named `file_index` and referred to the Windows test by a name identical to its own. - `save`'s doc comment said `Win32_Storage_FileSystem` was the only feature missing for the follow-up that moves `write_file_atomically` down into this crate. It is now present, so the note says the dependency side is ready and the remaining work is the move itself. Signed-off-by: Nowy <740902+nowycondro@users.noreply.github.com>
Restart onto a new runtime: - Split the pure half of a restart (load the record, apply the runtime pin, build the child argv) out of restart_internal_managed_service, and let restart_service_onto_runtime take the restart as a parameter, so a unit test reaches both call sites of the pin without launching an engine. - Count a restarted server under services_restarted only once it answered an inference request; a respawn that had not answered when the wait ended is reported under services_restarted_not_yet_serving and does not fail the command. Activation report: - Re-activating the active runtime prints changed_from_runtime_key: <unchanged> instead of naming the retained rollback target. - An unreadable services folder now says the switch was not made. Tests and scenarios: - Add the tests the review found missing: sdk-install finalization carries the reconciliation, the double-fault restore message, a rollback target whose runtime was uninstalled, the --yes/--restart-services clap rule, blank requested_runtime_id, and the exact-key fast path. - Fix the ordering test that could never fail. - Add scenarios for a bare --yes and for a refused restart on the mock lane; split the Then steps that ran the CLI; renumber scenarios after the rebase onto main, which already carries the re-selection scenario. Docs: use placeholder runtime keys, describe the three restart outcomes, fix the manual-testing walkthrough, shorten the config save comment. Signed-off-by: Nowy <740902+nowycondro@users.noreply.github.com>
bc30f4d to
2f004cd
Compare
Rebased onto current
|
clippy::redundant_clone flagged the clone of the install root, which is not used again after it moves into the rebased paths. Signed-off-by: Nowy <740902+nowycondro@users.noreply.github.com>
clippy (now gated on Windows) flags `&mut info` passed to a Win32 call as an implicit borrow as raw pointer; pass `&raw mut info`. Signed-off-by: Nowy <740902+nowycondro@users.noreply.github.com>
CI green at
|
jussielo-amd
left a comment
There was a problem hiding this comment.
Reviewed at head 7df9323 (merge-base with origin/main is 7a062ad, so the branch is cleanly rebased onto current main with no conflicts). This PR has an unusually long review history from an automated pr-review-watcher process across many rounds; the most recent CHANGES_REQUESTED review (at bc30f4d) raised roughly nine blocking items plus a merge-conflict/stale-rebase finding. Commit 2f004cd, pushed today, is a remediation pass that claims to address nearly all of them, followed by two trivial clippy fixes (9bcca96, 7df9323).
I independently re-verified each previously-blocking item against the current diff rather than trusting the commit message, and they check out as genuinely fixed:
- The branch no longer merges / duplicates a scenario already on base: resolved by the rebase itself — merge-base now equals origin/main HEAD, and
runtime_lifecycle.feature's scenario ids are unique within the file. - The restart-pin fix was revertible with every unit test green:
apps/rocm/src/main.rs:15902-15935now splitsload_service_for_restart/restart_child_argvout ofrestart_internal_managed_service, andpinning_the_record_is_what_puts_the_new_runtime_in_the_restart_argv(main.rs:31110) drives the realrestart_service_onto_runtime_withcall chain with a failing stand-in, so dropping either theapply_restart_runtime_pincall or the pin argument now fails the test. - The sdk-install ordering test could never fail:
render_sdk_install_success_includes_service_reconciliation_section(main.rs:33580) now setsprevious_runtime_key: Some(...)and uses.expect(...)on the rollback-hint position, so the ordering assertion is live. - A restart that never becomes ready was still counted as success:
restart_service_onto_runtime_with(runtime_services.rs:502-505) now only returnsServingwhen status isready; anything else isNotConfirmedand surfaces separately asservices_restarted_not_yet_serving. - Re-activating the active runtime printed a false
changed_from_runtime_key:changed_from_runtime_key_text(main.rs:6145) now reads areactivatedflag and prints<unchanged>on that path. - Blank
requested_runtime_idnot falling through toruntime_id:crates/rocm-core/src/lib.rs:4273-4278now uses.find(|value| !value.trim().is_empty())across both keys. - Missing
--yes-without---restart-servicescoverage and missing scenarios for a bare--yesand a refused mock-lane restart: present as@id:runtime-lifecycle-yes-without-restart-services-refusedand@id:runtime-lifecycle-failed-service-restart-is-reportedinruntime_lifecycle.feature. - The double-fault restore message and the uninstalled-rollback-target marker-id case both now have dedicated, non-vacuous tests (main.rs:31636, and the marker-snapshot tests nearby).
- CI is green across the board at this head (clippy, build-and-test, windows-build-and-test, Test (affected crates), E2E tests, coverage, commit-signatures), and
docs/architecture.mdnow listsruntime_services.rsas a full domain extraction, closing the "new subsystem grown inside main.rs" finding from an earlier round.
A few low-severity, non-blocking observations remain, none of which are correctness bugs or gaps introduced fresh by this PR's own diff against a clean baseline:
- The exact-match fast-path arm in
classify_service_runtime_state(runtime_services.rs:157) is no longer provably dead in isolation —a_record_naming_the_active_runtime_matches_without_any_manifest_to_resolve_itexercises it with an empty manifest list — but both real production call sites (activate_runtimeandrollback_runtime, viareconcile_services_for_runtime) always pass the full installed-manifest list, which already contains the activation target, so the arm stays unreachable along every live code path. Worth either deleting it or being explicit in the comment that it exists only as a defensive fallback for hypothetical partial-list callers. rollback_runtime's printedchanged_from_runtime_key(main.rs:5916) readsresult.previous_runtime_keydirectly rather than going through the newchanged_from_runtime_key_texthelper used on the activate path. Harmless today since rollback never reactivates, but the fix is one level shallower than the type it was meant to guard, so a future rollback-side no-op path could silently regress.activate_runtime/rollback_runtime's printed report (main.rs:5860 onward) is still hand-rolled withprintln!, and the two functions duplicate a fair amount of reconcile/snapshot/restore plumbing. This predates the PR (the same pattern exists on origin/main before this change), so it's a pre-existing-code follow-up rather than something this diff introduced, but centralizing it — perhaps onto the sharedcli_report.rs::ActionReport— would have saved this remediation round from needing two-site coordination.
Nothing here blocks. The transactional core (snapshot/restore, atomic writes, marker-vs-config ordering) and the reconciliation logic both hold up under the mutations I traced, the regression tests are non-vacuous, and the behavior changes are covered by Gherkin scenarios rather than unit tests alone. Approving.
Summary
rocm runtimes activate(androllback, and the--activatetail ofrocm updateandrocm install sdk) was transactional in neither of the two senses it needed to be.1. It never looked at the running services. The report ended with one fixed sentence — "running services keep their recorded runtime until they are restarted" — printed whether or not a single local server existed. It could not name a server that really was left behind, and it could not stay quiet when none was. There was no way to move those servers either.
2. It had no rollback across its two persisted writes. Activation writes
config.jsonand then the active-runtime marker. If the marker write failed, the config already named the new runtime while the marker still named the old one, and the marker feeds both runtime resolution and the storage retention holds.RocmCliConfig::savecompounded it: a barefs::writetruncates the liveconfig.jsonfirst, andloadhard-errors on a file it cannot parse.What changed
services_on_previous_runtime: N(always printed, including 0) with one line per stale server, andservices_with_unrecorded_runtimefor live records that name no runtime. Engines that manage their own runtime are never stale; anenv_idon a record is not a pin.--restart-services(with--yes, refused before anything is written without it) onactivateandrollbackmoves the stale servers onto the new runtime. Best effort per server; every failure is named and the command exits non-zero. Outcomes are reported apart:services_restarted(the server answered an inference request),services_restarted_not_yet_serving(respawned, had not answered within the 45 s wait; exit 0),services_restart_failed.load_managed_servicerefresh, which would otherwise re-adopt the outgoing process'sruntime_id/env_idand bring the server back on the runtime it was already using.ActivationSnapshotcaptures the config fields and the marker's bytes before the first write; a failed marker write restores both, in memory and on disk. Services are read before either write, so an unreadable services folder refuses the switch with the previous runtime fully in place (and the error says the switch was not made).RocmCliConfig::savewrites a sibling temp file and renames it overconfig.json.changed_from_runtime_key: <unchanged>.docs/testing.mdanddocs/manual-testing.mddescribe the flags and report.Risk: medium. Confined to the runtime-activation paths, but it adds a failure mode to them (an unreadable services folder now refuses activation, rollback,
install sdkfinalization andupdate --apply --activate) and--restart-servicesstops and respawns live servers.Test plan
Not compiled or run locally: the environment this branch was prepared in has no Rust toolchain and cannot install one, so CI (and the GPU lanes for scenario 12) is the first execution of the code added since the previous push.
rustfmtwas run (a wasm build of rustfmt with the repo's edition) over every changed Rust file and reports no diff.Scenarios in
tests/e2e-cucumber/features/runtime_lifecycle.feature(all on the mock lane except 12):runtime-lifecycle-08(already onmain) — re-selecting the active runtime keeps the rollback target; this PR adds the<unchanged>report line.10activation names the servers left on the previous runtime and stays silent when none is;11--restart-servicesis refused without--yes;13an unreadable service record refuses the activation and changes nothing;14the same refusal forrollback;15a failed marker write rolls the config back;16a server recording no runtime is counted apart;17a bare--yesis refused on both commands;18a refused server restart is reported, the switch stands, and the record is not rewritten.12— the--restart-services --yessuccess path.@requires-gpu @requires-engine:vllm: runs on the self-hosted MI300X and MI350P lanes, not on the hosted mock lane (a restart stops an engine and waits on a real readiness probe).Unit tests cover the pin reaching the restart argv through both production call sites, the three restart outcomes, the double-fault restore message, a rollback target whose runtime was uninstalled, the
--yes/--restart-servicesclap rule,install sdkfinalization carrying the reconciliation, and blankrequested_runtime_idhandling.Known residuals
rocm services restart <id> --yesstill replays the recorded runtime, so restarting a server by hand after a switch brings it back on the old runtime. Changing that is a user-visible semantic change for every restart, proposed as a follow-up; it is why the report's note namesrocm runtimes activate <key> --restart-services --yes.env_root_for_service) resolves the activated runtime's manifest after the server has been stopped. A failure there is reported and the record restored, but only the GPU lane reaches it.install sdkfinalization andupdate --apply --activate, where the install itself has already succeeded. Treating an individual record that vanished between listing and reading as "skip", and reporting "installed but not activated" on those two paths, is worth a maintainer's opinion; it is not changed here.env_idcannot distinguish a user pin from engine-reported state (both engines write it on every launch), so an explicit--env-idis not honoured across a runtime switch and--restart-servicesclears it. The symmetric fix (engines writingrequested_env_id) changes a contract across both engines.render_runtime_service_reconciliationdoes not go throughcli_report.rs::ActionReport, which models flat pairs; extending it is its own change.RocmCliConfig::saveuses a plainfs::rename, so aconfig.jsonheld open without delete sharing makes the save fail with a sharing violation (the marker write goes throughReplaceFileW). Movingwrite_file_atomicallyintorocm-coreis a separate change.Win32_Storage_FileSystemis enabled forrocm-coreon Windows only for a#[cfg(windows)]test;apps/rocmalready enables it, so there is no practical cost.AGENTS.md§11; the platform this work runs on requires it. A maintainer call.Commit signing
Commits are created through the GitHub API so they carry GitHub's verified signature and a
Signed-off-bytrailer.Ticket: EAI-7404
🤖 by agent-hub on AMD AgentHub