Repository navigation
ROCMAI-887: make a managed-service recovery work from the embedded and the standalone daemon - #948
Conversation
…uns inside rocm `supervise` is a command of the `rocmd` binary only. The recovery restart re-executed the current executable with it, which is right for `rocmd run` but wrong for the daemon that `rocm daemon run` embeds: the current executable there is `rocm`, which answers `supervise` with an unrecognised-subcommand error, so the restarted supervisor died at once and the service stayed failed. The restart now asks a small `DaemonHost` for the binary that has the command, before it touches the record. The embedded entry point, `run_from_args`, marks the process explicitly, so the choice does not depend on what the binary is called (an upper-case or versioned `rocm` is still `rocm`). Embedded, the restart runs `supervise` on the `rocmd` installed beside `rocm` and, when there is none, says what it was looking for and leaves the record exactly as it found it instead of marking it recovering for a supervisor that was never started. Otherwise the current executable is used, as before, so the standalone daemon and the unit tests that stand the test binary in for `rocmd` are unchanged. The choice and the sibling lookup start from one executable path, through a new `rocm_core::sibling_binary_path_from`, so they cannot disagree about it. This makes the restart start the right supervisor. Whether the engine that supervisor then launches starts is a separate change. Tests cover the choice for each executable name, the name and starting path the sibling lookup is given, the marking by `run_from_args` (in a child run of the test binary, since the flag is process-wide), a restart on a host that is `rocm` that runs `supervise` on a stand-in `rocmd` and records its arguments and supervisor, and one with no `rocmd` that changes nothing. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…om the rocmd daemon A supervisor started the engine by re-executing its own executable with the hidden `__engine-serve-http` command. That command belongs to the `rocm` binary, and `rocmd` answers it with an unrecognised-subcommand error, so after a recovery restart the supervisor was running but its engine never started: the service log held the clap error, the record kept saying `running` with a dead engine PID, and the daemon's watcher restarted it again and again. It affected the standalone daemon as much as the embedded one, because the restart always starts the supervisor as `rocmd`. `DaemonHost` gains the mirror of the choice made for the restart: `rocm_binary` returns the current executable when the daemon is embedded in `rocm`, and the `rocm` installed beside `rocmd` when it is not, naming what was missing when there is none. `supervise_service` takes the engine's binary from it, before its first write, so a supervisor that cannot find `rocm` leaves the service's record as it was, and the lookup and the choice start from the same executable path as the restart's do. `rocmd` still does not depend on the `rocm` crate; the binary is found on disk. The supervise tests that reach the engine spawn now run on a host whose own executable stands in for `rocm`, as the test binary stood in for it before. Tests cover the choice for each executable name and the name and path the lookup is given, a supervisor run by `rocmd` that starts its engine as `rocm __engine-serve-http <engine> <service>` on a stand-in `rocm`, and one with no `rocm` beside it that changes nothing and says what it looked for. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The daemon asked an engine a question, its readiness check or its stop, by re-executing its own executable with the hidden `__engine-stdio` command. That command belongs to the `rocm` binary, so run as `rocmd` the request failed with an unrecognised-subcommand error every time: the supervisor's readiness wait could never see its engine become ready, and a stop of a container-backed engine, whose confirmation depends on the engine's own answer, could never be confirmed. `engine_request` now runs the command on the binary `DaemonHost::rocm_binary` returns, the same choice the supervisor makes for its engine, so the current executable is used when the daemon is embedded in `rocm` and the `rocm` installed beside `rocmd` otherwise, with an error that names it when there is none. Tests run the request against a stand-in `rocm` that records its arguments, reads the request and answers, and against a `rocmd` with no `rocm` beside it. The script-writing helper and lock the earlier script tests carried now live with the other test support, and those tests use them. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…binaries The supervise tests that stood in for `rocm` ran the supervisor on a thread that went on polling the engine's readiness through the real host, so in a workspace-wide test run with `target/debug/rocm` built, that thread could run the real `rocm __engine-stdio`. The readiness poll now goes through the host the supervisor started its engine on, and `DaemonHost::detect`, the real host, is refused under test: a test passes the code it exercises a host of its own, and anything that reaches the real one fails at once instead of running a binary. The supervise test's stand-in `rocm` now also answers the readiness check and lets its engine exit, so the supervisor finishes by itself and the test joins it, and the restart tests name the test binary as their `rocmd` rather than relying on the real lookup. The stand-in scripts no longer assume `/bin/sleep`: their shell and `sleep` are found on `PATH` once and asserted by name, as is the `setsid` the restart runs. A test that polls through the real host, one that polls through a different host than the engine ran on, and one that lets `detect` run under test each fail. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
… spawn runs `docs/testing.md` lists the recovery and supervise tests but not the ones for choosing the binary, and nowhere describes that the daemon's spawns run on `rocmd` or `rocm` depending on the command and on where the daemon is running. It now lists the new tests and says which binary the restart, the engine start, the readiness check and the engine stop each run on, and how the unit tests keep off the real binaries. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🟡 Changes recommended
Missing Gherkin coverage through both daemon entry points leaves marker-detection regressions that could break recovery, readiness, and subsequent shutdown unverified.
1 open issue
What changed in this PR
This PR corrects executable selection during managed-service recovery so embedded and standalone daemons run commands on the binary that supports them.
Changes:
- Introduces
DaemonHostto distinguish embedded execution and selectrocmorrocmd. - Routes supervisor launches, engine starts, and engine requests through that selection, resolving missing binaries before record writes.
- Adds isolated regression tests and documents the routing and verification approach.
| File | Description |
|---|---|
| docs/testing.md | Documents targeted tests and daemon-host routing. |
| docs/architecture.md | Describes DaemonHost and its integration. |
| crates/rocm-core/src/lib.rs | Exports the explicit-path sibling lookup. |
| crates/rocm-core/src/binary_paths.rs | Adds sibling lookup from a supplied executable path and tests. |
| apps/rocmd/src/test_support.rs | Adds helpers for stand-in executable scripts. |
| apps/rocmd/src/supervise.rs | Selects supervisor and engine binaries; adds routing regression tests. |
| apps/rocmd/src/lib.rs | Registers the daemon-host module. |
| apps/rocmd/src/engine_ipc.rs | Routes engine requests through rocm and tests selection failures. |
| apps/rocmd/src/daemon_host.rs | Implements host-aware binary selection and unit-test isolation. |
| apps/rocmd/src/cli.rs | Marks the embedded daemon entry point. |
Review effort: Balanced
…t helper from being dead code on Windows The docs and three code comments advised `rocm daemon run`, which is not a command: the embedded daemon is `rocm daemon`, which takes only `--status`. The repository's own check that parses every advised command failed on both doc lines, and so did the jobs that run it. The helper that stands a test binary in for `rocmd` is used only by Linux-gated tests, so a Windows build with warnings denied reported it as never used; it now allows dead code on other platforms. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Human runtime validation is needed because stand-in tests do not establish end-to-end recovery, readiness, and stop behavior across the two executables or Windows subprocess behavior.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
The collision-safe serving ports change rewrote the recovery restart in `supervise.rs` (the launch lock, the re-read of other services' records for a claimed port, the loopback port lease, and the rollback of the record on a failed write or spawn), which this branch also changes to choose its binary through `DaemonHost`. The resolution keeps that restart as it is on main and adds the host choice to it: the `rocmd` binary is resolved first, before the launch lock is taken and before anything is read or written, in a host-taking core that `restart_managed_service_with` and `restart_managed_service_bounded` reach through `DaemonHost::detect`, so their signatures and the tests that use them are unchanged. Under test, `detect` is the test binary standing in for both commands, so those tests still run only the test binary. A new test checks the two together: a restart from `rocm` releases the leased port and then spawns the sibling `rocmd`. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Recovery spans multiple processes, but stand-in tests leave the installed-binary recovery, readiness, and stop lifecycle and Windows supervision unvalidated, requiring maintainer runtime validation.
No open issues 🎉
Review effort: Balanced
r0x0r
left a comment
There was a problem hiding this comment.
Reviewed at 3eaf9d7f. This is a good fix and I think it should go in — approving with three nits below, none of which block merge.
The thing I like most is the shape: the production change is about twenty lines across three call sites, all sitting behind one small, well-documented abstraction, and the root cause is fixed at the layer where both hosts actually share it rather than patched at each caller. I went looking for the usual failure mode in a change like this — a half-written record when the binary can't be found — and it isn't there: both failure paths resolve the binary before any write or any lock is taken, and each one has a test asserting the record is byte-identical afterwards.
Test quality is better than I expected too. Every branch of rocmd_binary_via / rocm_binary_via is pinned by a test that panics from inside the injected closure if the wrong branch is taken, so these are genuine mutation checks rather than happy-path assertions that would pass either way.
I also checked the origin/main merge, since that's where this kind of refactor usually loses something quietly. Diffing restart_managed_service_bounded as it stands on main against restart_managed_service_via here, the only delta is the moved binary resolution — main's launch lock, the port re-read and the lease rollback all survive intact. And all 38 checks are green on the reviewed head (36 passing, 2 skipped).
Two things for after this merges, neither of which changes my verdict:
sandbox.rs:77 is worth a ticket. The PR body describes it as "the same class of defect and unconfirmed." It's confirmed: apps/rocm/src/cli.rs:239 gives rocm only sandbox-run and no sandbox-tool, so the embedded daemon's bubblewrap path is in fact broken. This PR's claim still holds — recovery doesn't route through it, since server_recover.rs:233 calls restart_managed_service in-process — but it's a real defect rather than a speculative one, so it deserves its own ticket instead of a note in a PR body.
No Gherkin scenario for a user-observable change (AGENTS.md §3). The reason given is actually argued rather than just asserted — the harness has no fake engine for managed supervision, so a restart would start a real engine and leave a supervisor waiting minutes — which I find reasonable. Just confirm that's still the plan and not something that slipped.
| - [`sandbox_tool.rs`](../apps/rocmd/src/sandbox_tool.rs) — the sandbox-tool vocabulary: the `SandboxToolArg` value enum naming each sandbox tool, and `SandboxToolPolicy`, the artifact-download policy built from the `--allow-*` flags (including the Hugging Face token read from the environment when authenticated downloads are allowed). It uses no other module of the crate, so modules that need the vocabulary do not import [`cli.rs`](../apps/rocmd/src/cli.rs). | ||
| - [`service.rs`](../apps/rocmd/src/service.rs) — managed-service stop: terminating the recorded processes of a service after verifying their identity, and the stop report (`stop_managed_service`, `stop_report_confirmed`) that the sandbox and MCP tools read. | ||
| - [`supervise.rs`](../apps/rocmd/src/supervise.rs) — launching and re-launching a managed service's supervisor: `supervise_service` (the `rocmd supervise` entry point: launch the engine child, record its identity, wait for readiness, record the exit) with the serve-log startup-phase polling and readiness wait it uses, and `restart_managed_service` (the recovery respawn: re-launch `rocmd supervise` detached for a service and record the new supervisor). Writes after a supervisor's first go through the private `update_supervised_record`, which re-reads the record and leaves a stop marker or another supervisor's record alone. Its production code uses only [`common.rs`](../apps/rocmd/src/common.rs), [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs) and [`persistence.rs`](../apps/rocmd/src/persistence.rs). [`cli.rs`](../apps/rocmd/src/cli.rs) calls `supervise_service`; [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) (the server-recover handler) and [`sandbox.rs`](../apps/rocmd/src/sandbox.rs) (the `restart_server` tool) call `restart_managed_service`. | ||
| - [`supervise.rs`](../apps/rocmd/src/supervise.rs) — launching and re-launching a managed service's supervisor: `supervise_service` (the `rocmd supervise` entry point: launch the engine child, record its identity, wait for readiness, record the exit) with the serve-log startup-phase polling and readiness wait it uses (its engine child runs on the `rocm` binary that `DaemonHost::rocm_binary` finds, before the first write, since `rocmd` has no engine commands, and it polls readiness through the same host), and `restart_managed_service` (the recovery respawn: re-launch `rocmd supervise` detached for a service and record the new supervisor; the binary that runs `supervise` comes from [`daemon_host.rs`](../apps/rocmd/src/daemon_host.rs), and a restart that cannot find it fails before it writes the record). Writes after a supervisor's first go through the private `update_supervised_record`, which re-reads the record and leaves a stop marker or another supervisor's record alone. Its production code uses only [`common.rs`](../apps/rocmd/src/common.rs), [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs) and [`persistence.rs`](../apps/rocmd/src/persistence.rs). [`cli.rs`](../apps/rocmd/src/cli.rs) calls `supervise_service`; [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) (the server-recover handler) and [`sandbox.rs`](../apps/rocmd/src/sandbox.rs) (the `restart_server` tool) call `restart_managed_service`. |
There was a problem hiding this comment.
This entry still ends with "Its production code uses only common.rs, engine_ipc.rs and persistence.rs", but supervise.rs:5 now carries use crate::daemon_host::DaemonHost; in production code — so daemon_host.rs belongs in that list.
Worth noting that the parallel claim one bullet up, on engine_ipc.rs (line 141), was updated to say "but daemon_host.rs"; this one just got missed in the same pass. Since check-architecture-doc reads none of the prose, AGENTS.md §5 puts entry accuracy squarely on the author.
There was a problem hiding this comment.
Thanks, fixed in 989b312. The supervise.rs entry now lists daemon_host.rs among the modules its production code uses, and it also lists stop_requested from watchers.rs, which restart_managed_service_via calls and the entry had left out as well. I checked the neighbouring entries: engine_ipc.rs already names daemon_host.rs and is the only other module that imports it besides cli.rs, which only marks the process.
On the two points in the review body, which have no thread of their own. The sandbox.rs defect: I confirmed it from the code and from a built binary. rocm sandbox-tool answers unrecognized subcommand 'sandbox-tool' (it suggests sandbox-run), and run_bubblewrap_sandbox re-executes current_exe with that verb, so it is the same class of defect. I could not find a path that reaches it from the embedded daemon today: the embedded entry runs a fixed rocmd run --automations-enabled, the watchers call run_sandbox_tool in process, and the only caller of the bubblewrap runner is the rocmd sandbox-run command, where current_exe is rocmd. So I would call it latent and not broken; I have not filed a ticket for it.
The Gherkin scenario: that is still the plan and nothing has slipped. The fixture binaries on current main are rocm-demo-env, fake-tailscale, fake-uv, fake-python and fake-docker, there is no fake engine, and no scenario or step definition references __engine-stdio or __engine-serve-http. The service-stop scenarios stage records and call rocmd sandbox-tool stop_server, and serve-41 fabricates a launch with no process behind it, so a restart through supervise would still start a real engine. The PR text says so.
| `rocmd`'s unit tests never reach the real host: `DaemonHost::detect` refuses to | ||
| run under test, each test passes the code under test a host of its own, and a | ||
| test that needs a binary writes a stand-in script (its shell and `sleep` found | ||
| on `PATH` once and asserted by name), so no `rocm` that happens to sit in the | ||
| build directory can run. `run_from_args_marks_the_process_as_embedded_in_rocm` |
There was a problem hiding this comment.
This paragraph and the daemon_host.rs entry at docs/architecture.md:140 both overclaim the test isolation, in the same two ways.
DaemonHost::detect doesn't "refuse to run under test" (docs/architecture.md:140 words it as "is refused") — the #[cfg(test)] arm returns a stand-in host, it doesn't refuse.
And "no rocm that happens to sit in the build directory can run" isn't enforced for test_binary_standing_in_for_rocmd() at apps/rocmd/src/daemon_host.rs:150: calling rocm_binary() on that host would resolve target/debug/rocm and execute it. Not reachable today, since only rocmd_binary() is ever called on it — but it's exactly the hazard commit 46e6227 closed for detect(), so the guarantee as written is stronger than what the code gives. Worth rewording both spots to match.
There was a problem hiding this comment.
Thanks, you are right on both counts, and both places are reworded in b056872. DaemonHost::detect does not refuse under test: it returns the test binary for both commands and looks for no sibling, and the architecture entry and the paragraph in the testing guide now say that. The hazard you found in test_binary_standing_in_for_rocmd was real as well, since asking that host for rocm_binary() would have run the sibling lookup and could have resolved target/debug/rocm, so I closed it in 1639f9b instead of only rewording: both stand-in hosts now look for no sibling and fail when asked for the binary they do not stand in for, and a_stand_in_host_looks_for_no_sibling pins both directions (it fails when the refusal is removed). A search of the docs and code comments found no other place with the same wording.
| #[cfg(not(test))] | ||
| { | ||
| let current_exe = rocm_core::current_executable_path() | ||
| .context("failed to resolve the current executable path")?; | ||
| Ok(Self::new(current_exe, is_embedded_in_rocm())) |
There was a problem hiding this comment.
The production body of detect() is #[cfg(not(test))], so this Self::new(current_exe, is_embedded_in_rocm()) wiring is compiled out of every unit test, and no e2e covers it either. The two binary choices downstream are covered well, but an inverted flag here would be caught by nothing — and it's the premise the whole fix rests on.
Injecting the flag, or reading it through a thread-local override, would make detect()'s own composition reachable from a test.
There was a problem hiding this comment.
Thanks, fixed in a205a4f. detect() now composes the host the same way in both builds, and only how the executable is found and the test stand-in marker differ, so the line that takes the embedded flag from the process is compiled into the tests. The flag is read through a thread-local override under test, which leaves the process-wide flag alone for every other test, and detect_takes_the_embedded_flag_from_the_process pins both answers; inverting the flag at that line makes it fail. What a unit test still cannot reach is the production arm that resolves the executable with rocm_core::current_executable_path.
…architecture entry The entry said supervise.rs's production code uses only common.rs, engine_ipc.rs and persistence.rs. It also imports DaemonHost from daemon_host.rs, and restart_managed_service_via calls stop_requested from watchers.rs, so both belong in the list. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…tand in for test_binary_standing_in_for_rocm and test_binary_standing_in_for_rocmd each name only one binary, and asking either for the other would run the real sibling lookup, which finds a rocm or rocmd built beside the test binary and lets a test run it. Nothing reaches that today, but it is the hazard the detect() stand-in was written to remove. The two stand-ins now look for no sibling, so the other binary is an error that never starts a lookup; a test pins both directions. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
DaemonHost::detect does not refuse to run under test: under test it returns the test binary for both commands and looks for no sibling. The two stand-in hosts name only the binary they stand in for and fail when asked for the other. Reword the daemon_host.rs entry in docs/architecture.md and the paragraph in docs/testing.md to match. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…ed flag The production body of DaemonHost::detect was compiled out of every unit test, so an inverted or constant embedded flag, which decides which binary each spawn runs, was caught by nothing. detect() now composes the host the same way in both builds, differing only in how the executable is found and in the test stand-in marker. A test reads the flag through a thread-local override, so the process-wide flag stays untouched for other tests, and pins both answers. The production arm that resolves the executable is still not reachable from a unit test. Signed-off-by: Jussi Elo <jussi.elo@amd.com>

This makes a managed-service recovery work from both the embedded and the standalone daemon.
superviseis a command ofrocmdalone, and the hidden engine commands arerocm's alone, but the daemon re-executed its own executable for each. Embedded inrocm, the restart ranrocm supervise, which fails. Standalone, the supervisor and its readiness and stop checks ranrocmd __engine-serve-httpandrocmd __engine-stdio, which also fail, leaving the servicefailedand restarted forever.A small
DaemonHostnow chooses the binary.run_from_argsmarks the embedded case, so the name of the binary does not matter, and each choice and sibling lookup start from one executable path. The restart, the supervisor's engine, its readiness poll and the engine requests run on the right binary, and a missing sibling is named. The restart and the supervisor resolve it before writing, so the record is left as it was. Unit tests do not reach the real host: under testDaemonHost::detectreturns the test binary for both commands and looks for no sibling, the two stand-in hosts name only the binary they stand in for and fail when asked for the other, and a test takes the embedded flag through a thread-local override sodetect's own composition is covered. The restart resolves the siblingrocmdbefore it takes the managed-launch lock, so a missing binary fails without holding the lock or touching a record, and a test shows that a restart fromrocmreleases its leased port before it spawns thatrocmd.User-visible change: after a recovery the service's engine starts, and its readiness and stop are seen. There is no Gherkin scenario, because the harness has no fake engine for managed supervision and a restart would start a real engine and leave a supervisor waiting for minutes. Unit tests with stand-in binaries cover each choice, and each was shown to fail when its branch is broken. Run locally: the
rocmdtests, clippy with-D warningsonrocmdandrocm-core, fmt and the crate-edge check. Not run locally: the full workspace gate, Windows and an end-to-end daemon recovery.Follow-ups, not part of this change:
supervise_servicerunsrocm __engine-serve-httpwithout the--log-path,ROCM_CLI_ENGINE_ENVS_ROOTand runtime-id and env-id handling thatrocm servepasses, and does not detach the engine child from the supervisor, so a standalone-daemon restart of a runtime-id service may not resolve the same engine environment.run_bubblewrap_sandboxinsandbox.rsstill re-executescurrent_exewith therocmd-onlysandbox-toolcommand, whichrocmdoes not have (rocm sandbox-toolanswersunrecognized subcommand), so it is the same class of defect. It is not reached from the embedded daemon today: the embedded entry runs a fixedrocmd run --automations-enabled, the watchers callrun_sandbox_toolin process, and the only caller of the bubblewrap runner is therocmd sandbox-runcommand, wherecurrent_exeisrocmd.