Skip to content

agent/73: serve: exit nonzero when the backend server dies in --foreground mode - #74

Merged
mikeroySoft merged 12 commits into
mainfrom
agent/73
Oct 25, 2026
Merged

mikeroySoft merged 12 commits into
mainfrom
agent/73

Conversation

@mikeroysoft-agent

Copy link
Copy Markdown
Collaborator

Closes #73

Gate report

Gate report

Host toolchain

ROCm: unavailable
Kernel: 7.2.5-3-omarchy
amdgpu: unavailable

  • conflict-markers: PASS
  • clippy: PASS
  • tests: PASS
  • smoke: PASS
  • leak-scan: PASS

@mikeroysoft-agent

Copy link
Copy Markdown
Collaborator Author

I found one blocking issue: when a vLLM backend is killed by a signal, the signal is never logged, so the logging acceptance criterion is only met for Lemonade. Everything else meets the issue and the repo rules. I didn't run any builds or tests, per the instructions, and I couldn't fetch issue #73 through gh because it isn't authenticated. I used the issue text and agent brief copied into .factory-prompt.md, which also contains the issue comments.

Root cause. The fix addresses the actual cause. On Linux, rocm_core::process_is_running uses kill(pid, 0), which returns true for a child that has exited but hasn't been reaped yet (a zombie). The old loop never reaped the supervisor child, so it waited forever. The new code keeps the Child handle and polls it with try_wait. The supervisor really does exit when its backend dies: Lemonade's serve_direct_llama_server calls bail! at engines/lemonade/src/process.rs:589. That is the "Error: …" line in the incident journal.

Spec (issue #73)

Required fixes

  1. The backend's exit signal is not logged for vLLM.
    • Criterion: "The exit is logged with the backend's exit status/signal." The brief adds: "when the backend server dies for any reason… log the backend's exit status/signal."
    • What the diff logs: the line at apps/rocm/src/service_launch.rs:1178 and :1188 reports the supervisor's exit status. The backend's signal only reaches the output if the engine supervisor writes it to the log.
    • Lemonade: it does (engines/lemonade/src/process.rs:589, engines/lemonade/src/lib.rs:687).
    • vLLM: it doesn't. engines/vllm/src/process.rs:168 calls std::process::exit(status.code().unwrap_or(1)) and prints nothing.
    • Trigger: rocm serve <model> --engine vllm --foreground, then kill -9 <vllm pid>.
    • Impact: the output says "exited unexpectedly with status exit status: 1" and the SIGKILL appears nowhere.
    • Docs now false for vLLM: this diff adds the claim in README.md:715 ("prints its exit status or signal"), docs/manual-testing.md:289 ("prints the backend exit signal") and docs/testing.md:916. AGENTS.md §5 says each such claim must be checked against the actual code path.
    • Fix: log the status before exiting in engines/vllm/src/process.rs, matching Lemonade, e.g. eprintln!("vLLM server exited with status {status}") just before line 168.

Optional suggestions

  • The SIGKILL test checks a line the test fixture writes. tests/e2e-cucumber/tests/e2e/serving_steps.rs:1973 looks for signal: 9 (SIGKILL), but the shell fixture echoes that exact string itself (apps/rocm/src/service_launch.rs:539). The test does prove the log is drained after the supervisor exits. It does not prove any real engine logs its signal; AGENTS.md §3 says a wording-only check "certifies the string, not the truth of it." Saying so in the PR text, next to the fix above, is enough.
  • Killing a daemon other services may still use. service_launch.rs:1110 with :1029 stops the daemon on user-stop and on backend death whenever this invocation started it. The brief asks for this. Edge case: rocm serve A --foreground starts the daemon, a later rocm serve B (background) reuses it, then Ctrl-C on A kills the daemon that B relies on for health checks and recovery. Consider skipping the daemon kill when other managed services are still live.
  • SIGINT/SIGTERM before log streaming starts. The signal watcher only exists inside stream_attached_logs. A signal that arrives during startup still kills rocm serve with the default action and leaves the managed child running in its own session. This was already the case before the diff; under systemd's default cgroup kill it doesn't matter.

Standards (AGENTS.md / docs)

Required fixes

None.

Optional suggestions

  • A false comment justifies the new signal-hook dependency. apps/rocm/src/service_launch.rs:912 says signal-hook "restores its registrations when the handle closes." It doesn't. The signal-hook-registry 1.4.8 unregister docs say it "does not currently return the default/previous signal handler," so SIGINT/SIGTERM are silently ignored for the rest of the process after the watcher is dropped. In practice that's only the cleanup window in stop_attached_service, where a second Ctrl-C or SIGTERM does nothing for up to about 6 seconds. Correct the comment. The crate was already in Cargo.lock through crossterm, so no new download.
  • Doubled wording in the exit message. service_launch.rs:1178 and :1188 print with status exit status: 1, because ExitStatus's Display already starts with "exit status". The error then repeats in main's Error: line.
  • Comments removed that were unrelated to the fix. The old comment explaining why raw mode is entered before the key-reader thread starts (it closes a window where Ctrl-C would kill the CLI) is gone above service_launch.rs:1273. So are the lock-release rationale at :1080 and the docs on RawModeGuard and detach_key_outcome. These described real constraints; consider restoring them.
  • Commits that aren't about serve: exit nonzero when the backend server dies in --foreground mode #73. comfyui.rs, therock.rs, system_sdk.rs (test-only) and crates/rocm-core/src/uv.rs (a production error-message change) were gate blockers, according to .factory/handoff-73.md. AGENTS.md §11 asks for one logical change per PR, so either explain them in the PR body or split them out.

Already conformant:

  • Scenarios: serve-27 and serve-28 have @id: tags and use tags that exist (@requires-no-gpu, @requires-os:linux).
  • Docs: --help, README, docs/testing.md and docs/manual-testing.md were all updated together.
  • Links: the new link in docs/testing.md is relative.
  • Test-only hook: it is behind cfg(all(feature = "e2e-test-hooks", unix)), and the release build gets a const fn that returns None.

Totals: Spec has 1 required fix and 3 optional suggestions. The worst is that a vLLM backend's kill signal isn't logged anywhere, which the new docs claim it is. Standards has no required fixes and 4 optional suggestions. The worst is the false signal-hook comment: the code claims signal handling is restored when, after the watcher is dropped, SIGINT/SIGTERM are ignored.

VERDICT: REVISE

@mikeroysoft-agent

Copy link
Copy Markdown
Collaborator Author

No required fixes remain, so this is an APPROVE. Round 1's one blocking finding (vLLM never logged the backend's signal) is fixed at engines/vllm/src/process.rs:168, and the diff now meets every required acceptance criterion in issue #73. I reviewed the diff only and ran no builds or tests. gh isn't authenticated, so I took the issue text, agent brief and comments from .factory-prompt.md.

Spec (issue #73)

Criterion Status Evidence
SIGKILL the backend → rocm serve --foreground exits nonzero within a few seconds, with no orphaned backend or daemon Met The old loop never reaped the supervisor child. The new code keeps the Child and polls it with try_wait (apps/rocm/src/service_launch.rs attached_process_exit). On exit it cleans up, then bail!s (:1181-1188). Scenario serve-27 checks the exit, the 5 s bound, and that the backend, supervisor and daemon are gone (tests/e2e-cucumber/tests/e2e/serving_steps.rs:1944, assert_foreground_processes_cleaned).
A user stop (SIGTERM/SIGINT) still exits 0 Met AttachSignalWatcher turns the signal into AttachOutcome::Stop, which cleans up and returns Ok(()). Scenario serve-28 covers it (serving_steps.rs:1958).
The backend's exit status/signal is logged Met Lemonade's supervisor already printed it. vLLM now does too (engines/vllm/src/process.rs:168). The CLI prints the supervisor's status (service_launch.rs:1178).
Optional --restart-backend N Not done Marked optional and off by default in the issue, so this is acceptable.

Required fixes: none.

Optional suggestions:

  • The SIGKILL log check matches text the fixture writes itself. The fixture echoes the exact signal: 9 (SIGKILL) string that assert_backend_sigkill_logged looks for. The check proves the log is still drained after the supervisor exits. It doesn't prove a real engine logs its signal. Say so in the PR text.
  • The 5 s bound includes startup time. started is taken before the spawn (serving_steps.rs:1827), so elapsed covers serve startup plus cleanup, not just the time after the kill. On a slow CI runner this could flake. Timing from the moment the signal is sent matches the criterion more closely.
  • A clean exit 0 from the supervisor is also treated as a failure. At service_launch.rs:1188 that happens, for example, when someone runs rocm services stop <id> from another shell. The issue says "for any reason", so this is in spec. It does mean systemd's Restart=on-failure would restart a service that was deliberately stopped that way.

Standards (AGENTS.md / docs)

Required fixes: none.

Optional suggestions:

  • A comment still claims signal handling is restored, and it isn't. service_launch.rs:912 says signal-hook "restores its registrations when the handle closes". In signal-hook-registry 1.4.8, unregister only removes the action from the registry and never reinstalls the default handler. After the watcher is dropped, SIGINT/SIGTERM are ignored for the rest of the process. In practice that's only the cleanup window in stop_attached_service, bounded at roughly 6 s plus the sandbox stop, before the process exits. The comment should still be corrected. Round 1 raised this and it is unchanged.
  • The exit message repeats its wording. service_launch.rs:1178 and :1188 print with status exit status: 1, because ExitStatus's Display already starts with "exit status".
  • Comments unrelated to the fix were removed. Gone are the reason raw mode is entered before the key-reader thread starts (above :1273), the lock-release rationale (:1080), the reason for the AlreadyRunning branch, and the docs on RawModeGuard and detach_key_outcome. These described real constraints and are worth restoring.
  • Some commits aren't about serve: exit nonzero when the backend server dies in --foreground mode #73. comfyui.rs, therock.rs and system_sdk.rs are test-only. crates/rocm-core/src/uv.rs changes a production error message. AGENTS.md §11 asks for one logical change per PR, so name them in the PR body as gate-blocker fixes, or split them out.

What already conforms:

  • Scenarios: serve-27/serve-28 have @id: tags plus OS/GPU tags, and are named in docs/testing.md, as AGENTS.md §3 asks.
  • Docs: --help (main.rs), README, docs/testing.md and docs/manual-testing.md were updated together (§5), and the vLLM fix makes their logging claims true.
  • New link: the new link in docs/testing.md is relative.
  • Test hooks: they sit behind cfg(all(feature = "e2e-test-hooks", unix)), the same pattern as the existing ROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE hook. Production builds get a const fn that returns None.
  • New dependency: signal-hook was already in Cargo.lock through crossterm. The watcher it supports is needed for the SIGTERM → exit 0 criterion, because without a handler SIGTERM's default action kills the process with no cleanup.

VERDICT: APPROVE

@mikeroysoft-agent mikeroysoft-agent added factory-approved Reviewer APPROVE recorded by the factory; merge-stage precondition and removed factory-approved Reviewer APPROVE recorded by the factory; merge-stage precondition labels Oct 22, 2026
@mikeroysoft-agent

Copy link
Copy Markdown
Collaborator Author

I found no required fixes, so the verdict is APPROVE. Since the previous APPROVE at b5b03a75, one commit has landed, 8eeaf862. It makes foreground_result a const fn (tests/e2e-cucumber/tests/e2e/serving_steps.rs:1932) and regenerates the dependency files MANIFEST.md:257 and THIRD_PARTY_NOTICES.txt. Neither change affects runtime behaviour.

I reviewed the diff only and ran no builds or tests. gh is not authenticated, so the issue text, agent brief and comments came from .factory-prompt.md.

Spec (issue #73)

Criterion Status Evidence
A test that SIGKILLs the backend shows rocm serve --foreground exits nonzero within a few seconds, with no orphaned backend or daemon Met The supervisor Child is now kept and polled with try_wait. The old process_is_running(pid) check saw the unreaped zombie as still running. On exit the code cleans up through stop_attached_service (apps/rocm/src/service_launch.rs:985), then fails with bail! (:1188). Scenario serve-27 checks the nonzero exit, the 5 s limit, and that the backend, supervisor and daemon are gone.
A user stop (SIGTERM/SIGINT) still exits 0 Met When a signal arrives, signal_watcher.requested() returns AttachOutcome::Stop (service_launch.rs:1311, :1341), which leads to cleanup and Ok(()). Scenario serve-28 covers SIGTERM.
The backend's exit status or signal is logged Met Lemonade already wrote it (engines/lemonade/src/process.rs:589). vLLM now writes it too (engines/vllm/src/process.rs:168). The log is read one final time after the exit is detected, so the last error line reaches the output.
Optional --restart-backend N Not done The issue marks it optional and off by default.

Required fixes: none.

Optional suggestions:

  • The SIGKILL log check matches text the fixture writes itself. The test looks for signal: 9 (SIGKILL) (serving_steps.rs:1973), and the fixture prints exactly that (service_launch.rs:539). So the check proves the final log read works, not that a real engine logs its signal. Say so in the PR text.
  • The 5 s limit includes startup time. started is taken before the spawn (serving_steps.rs:1827). On a slow runner this could fail even when the exit after the kill is fast. Measuring from the moment the signal is sent would match the criterion more closely.
  • Stopping the daemon on exit can affect other services. service_launch.rs:1028 stops the daemon whenever this invocation started it. If another background rocm serve began later and reused that daemon, it loses health checks and auto-recovery until some command restarts the daemon. The issue's "no orphaned daemon" requirement asks for this, but it is a tradeoff worth stating in the PR text.

Standards (AGENTS.md / docs)

Required fixes: none.

Optional suggestions:

  • The comment at service_launch.rs:912 is wrong. It says signal-hook "restores its registrations when the handle closes". The signal-hook-registry 1.4.8 docs on unregister say the opposite: the default handler is not reinstalled, so the program "will effectively ignore SIGTERM". After the watcher is dropped, SIGINT/SIGTERM are ignored during cleanup in stop_attached_service, which runs for up to about 6 s plus the sandbox stop. The impact is limited because the process exits right afterwards. This is the third round this has been raised and the comment is unchanged.

  • Windows can lose PID-reuse protection. On Windows child is None, and service_launch.rs:992 captures the supervisor's ProcessIdentity at stop time. The spawn-time identity already recorded at :582 is not used. If the supervisor exits and its PID is reused before that capture, terminate_verified(Tree, …) would match and kill an unrelated process tree. The window is small, but reusing the spawn-time identity would close it.

  • The exit message repeats "exit status". service_launch.rs:1178 and :1188 print with status exit status: 1, because ExitStatus's Display output already begins with "exit status".

  • Comments that describe real constraints were deleted. These are the reason for entering raw mode before the key reader starts, why the launch lock is released, why the AlreadyRunning branch exists, and the docs on RawModeGuard and detach_key_outcome. They are unrelated to the fix and worth restoring.

  • Some changes are outside the issue's scope. AGENTS.md §11 asks for one logical change per PR. The out-of-scope changes are:

    • the test fixes in comfyui.rs, therock.rs and system_sdk.rs;
    • the production error message change in crates/rocm-core/src/uv.rs;
    • the jsonc-parser 0.33.1 → 0.33.2 version bump in MANIFEST.md:257, which corrects drift already on main.

    Each was needed to get the gate and CI green (§12). List them in the PR body as gate fixes, or split them out.

What already follows the rules:

  • Scenarios: serve-27/serve-28 carry @id:, @requires-os:linux and @requires-no-gpu tags, and docs/testing.md names them with a relative link (§3, §5).
  • Docs: --help (apps/rocm/src/main.rs:472), README, docs/testing.md and docs/manual-testing.md were updated together (§5).
  • Test hooks: they are behind cfg(all(feature = "e2e-test-hooks", unix)), and production builds get a const fn that returns None.
  • New dependency: signal-hook was already in Cargo.lock as a dependency of another crate, and the regenerated MANIFEST.md and THIRD_PARTY_NOTICES.txt match the lockfile. A SIGTERM handler is needed for the "exit 0 on user stop" criterion, because SIGTERM's default action kills the process with no cleanup.

VERDICT: APPROVE

@mikeroysoft-agent mikeroysoft-agent added factory-approved Reviewer APPROVE recorded by the factory; merge-stage precondition and removed factory-approved Reviewer APPROVE recorded by the factory; merge-stage precondition labels Oct 22, 2026
@mikeroysoft-agent

Copy link
Copy Markdown
Collaborator Author

Spec check: issue #73

Criterion Status Evidence
Backend SIGKILL makes rocm serve --foreground exit nonzero within a few seconds, with no orphaned backend or daemon Met apps/rocm/src/service_launch.rs:709-720 keeps the supervisor's Child and checks it with try_wait. That fixes the root cause. Before, the dropped child became a zombie, and process_is_running uses kill(pid,0), which still succeeds for a zombie (snapshot crates/rocm-core/src/process.rs:371-383), so the old loop never saw the exit. service_launch.rs:664-678 cleans up, then returns an error. serve-27 (model_serving.feature:1267-1272, serving_steps.rs:1413-1526) would fail without the fix: the parent would hang until the 5 s timeout.
SIGTERM/SIGINT to rocm serve stops cleanly and exits 0 Met service_launch.rs:413-466 adds a signal watcher. service_launch.rs:795-797 and :851-853 handle stop requests, and :633-660 returns Ok. serve-28 covers this (model_serving.feature:1274-1278).
Exit is logged with the backend's status/signal Met service_launch.rs:667 prints the supervisor's status. service_launch.rs:823 and :870 read the rest of the engine log after the exit, so the backend's own error line gets printed. Lemonade already logs that line (see the journal in the issue). For vLLM, engines/vllm/src/process.rs:1251 now logs the ExitStatus.
Optional --restart-backend N Not done It is optional, so this is fine.
Docs kept in sync (AGENTS.md §5) Met README :53-58, --help in main.rs:161-173, docs/testing.md:1227-1238, docs/manual-testing.md:1199-1215.

Required fixes

  1. The diff deletes factory dispatch/manager config that serve: exit nonzero when the backend server dies in --foreground mode #73 has nothing to do with. review.diff:10-16 (.factory.toml, base lines 25-32) removes [dispatch] review_rounds = 3 and [manager] rounds = 3.
    • Rule broken: AGENTS.md §11, "keep each PR scoped to one logical change", and §13 item 9.
    • Effect: the snapshot (the gated revision) has no such sections. Landing the gated candidate as-is drops both settings back to defaults. The 2026-10-22T20:57 issue comment shows the default is 1 manager round, so the factory's review and escalation loop shrinks from 3 rounds to 1.
    • Fix: restore the two sections, or rebase so the PR does not touch .factory.toml. This may be leftover from an outdated base rather than something the worker did; I can't tell which from the diff. Either way, the candidate should not carry the deletion.

Optional suggestions (non-blocking)

  • rocm services stop on an attached serve now counts as a failure. service_launch.rs:664-678 treats every supervisor exit as unexpected, including one the user asked for through rocm services stop <id> (the stop path --help points to, main.rs:162-163). Under Restart=on-failure, systemd would then restart the unit right after an explicit stop, and the line "exited unexpectedly" is not true in that case. The issue's own wording ("for any reason") allows this behaviour, so it does not block. If it matters, check the record's stop_requested_unix_ms before failing.
  • Shared daemon gets killed. service_launch.rs:525-538 stops the daemon whenever this invocation started it, even if a later background rocm serve is now relying on it. That service then loses health checks and auto-recovery until some later CLI call restarts the daemon. Checking for other live managed records before stopping it would avoid this.
  • A simpler signal mechanism would do. AttachSignalWatcher (service_launch.rs:413-466, a thread, a channel and a Drop impl) could be replaced by signal_hook::flag::register(SIGINT|SIGTERM, Arc<AtomicBool>). That needs no thread and no join. The current version works and is needed for criterion 2; the flag version is just less code.
  • E2E runs a different path locally. serving_steps.rs:1425-1435 does not set stdin(Stdio::null()). Run from an interactive terminal, the child takes the TTY/raw-mode path (service_launch.rs:765-776) instead of the non-TTY path CI uses. If the run times out and child.kill() fires, the terminal can be left in raw mode.
  • Rationale comments deleted. Unrelated comments explaining non-obvious ordering were removed at review.diff:555-557, 587-592, 770-774 (why raw mode is entered before the key reader starts) and 804-806 (why \r\n is needed in raw mode). The code still depends on those orderings, so consider restoring them.
  • Weak logging assertion. serving_steps.rs:1577-1585 passes on the fixture's own echoed line (service_launch.rs:291). It only proves the final log read forwards the supervisor's stderr. The vLLM status line at engines/vllm/src/process.rs:1251 has no test.
  • More unrelated changes. crates/rocm-core/src/uv.rs:1169-1188 changes a production error message, and there are test-flake fixes in therock.rs:1117-1142, comfyui.rs:147 and system_sdk.rs:1157. Per the handoff these were gate blockers, but under §11 they would be cleaner as a separate PR.

VERDICT: REVISE

@mikeroysoft-agent

Copy link
Copy Markdown
Collaborator Author

Standards

Required fixes

  1. The diff deletes the step definition for serve-23, but the scenario is still in the feature file.

    • Evidence: review.diff:1349-1356 removes #[then("Ornith appears in the model list")]. The gated snapshot still has the scenario that uses it: @id:serve-model-list-ornith / Then Ornith appears in the model list (snapshot/tests/e2e-cucumber/features/model_serving.feature:280-283). The Ornith catalog entry is still there too (crates/rocm-core/src/model_catalog.json:447).
    • Impact: serve-23 now has an undefined step. Cucumber either skips it or fails it, so the Ornith listing check is gone either way. I found no fail_on_skipped in tests/e2e-cucumber, so the likely result is a silent skip.
    • Rule: AGENTS.md §1 says "Solve the reported problem, not an adjacent symptom." This hunk has nothing to do with serve: exit nonzero when the backend server dies in --foreground mode #73.
    • Fix: restore the step, most likely by rebasing onto the current base and dropping the hunk.
  2. The diff reverts the dash-35 terminal-width fix back to 80 columns.

    • Evidence: review.diff:1308-1314 removes the explanation that the hardware panel gives the GPU column only 3/5 of the width, so "at the standard 80 columns that column is ~48 wide and the vram segment is (correctly) dropped". review.diff:1320,1325-1326 replace resize_to(70, 120) with grow_rows(70), which keeps 80 columns. review.diff:1653-1654 makes resize_to private again.
    • The new comment at review.diff:1315-1318 says 80 columns is "comfortably over the 54" columns needed. The snapshot says otherwise:
      • hardware.rs:53 still splits the panel Ratio(3, 5).
      • observe.rs:60 renders that same panel.
      • The threshold test at hardware.rs:928-931 still pins 54 columns.
    • Impact: at 80 columns the compact GPU row is about 46 usable columns, so it drops the VRAM segment. The gpu_row.contains("GiB") assertion at dash_steps.rs:2168-2171 can no longer see the units the scenario exists to check.
    • The gate's "tests: PASS" doesn't show that dash-35 ran. The worker ran e2e separately and only for serve-(27|28).
    • Fix: rebase onto the current base and drop these hunks. Same rule as item 1 (AGENTS.md §1, outside serve: exit nonzero when the backend server dies in --foreground mode #73's scope).

Optional suggestions

  • The handoff doesn't explain these unrelated changes:

    • The jsonc-parser 0.33.1→0.33.2 bump (review.diff:9-13,33-34,66-67).
    • The rocm coverage floor change, where the measured line count drops from 58218 to 44823 (review.diff:1139-1140).

    Both may be artifacts of a stale base. Check them before merging.

  • The diff deletes explanatory comments that serve: exit nonzero when the backend server dies in --foreground mode #73 didn't require touching:

    • The reason raw mode is entered before the key reader starts (review.diff:755-759).
    • The reason the launch lock is released early (review.diff:540-542).
    • The reason the attached path starts the helper daemon (review.diff:572-577).

    The raw-mode one records a real ordering requirement and is worth restoring.

  • No new crate is added: signal-hook is already in the lock graph, and the diff only adds the edge at review.diff:21,79. AttachSignalWatcher (review.diff:398-451) is needed for the SIGTERM/SIGINT criterion. Neither is an issue.

Spec (issue #73)

The core criteria are met:

  • Root cause. The old code dropped the Child, so the dead supervisor stayed a zombie and process_is_running kept reporting it alive. The diff keeps the Child and calls try_wait on it (review.diff:239,296,698-704).
  • Nonzero exit after cleanup. An unexpected exit triggers cleanup and then bail! (review.diff:649-663).
  • Clean user stop exits 0. SIGINT/SIGTERM take the Stop path (review.diff:780-782,836-838).
  • Exit status is logged. The log is drained once more after the supervisor exits (review.diff:808,855), and vLLM now prints the backend's status before exiting (review.diff:1249).
  • Tests. serve-27 (SIGKILL the backend → nonzero within 5s, signal logged, no orphan backend/supervisor/daemon) and serve-28 (SIGTERM → exit 0, no orphans) are at review.diff:1265-1276.
  • Docs. The README, --help, docs/testing.md and docs/manual-testing.md are updated as AGENTS.md §5 requires (review.diff:46-54,146-158,1197-1236).
  • The optional --restart-backend N flag is not implemented, which the issue allows.

Optional suggestions

  • The SIGKILL log check matches text the fixture writes itself. serve-27 asserts signal: 9 (SIGKILL) (review.diff:1626), but that string comes from the test shell script (review.diff:275-276). It proves the final log drain passes the line through, not how the real Lemonade supervisor formats it. That is acceptable without a GPU, but say so in the PR text, as AGENTS.md §3 asks for coverage gaps.
  • The exit message repeats the word "status". ExitStatus's Display already starts with "exit status: …", so the output reads "exited unexpectedly with status exit status: 1" (review.diff:650-652,663).
  • A user stop now kills the daemon this invocation started (review.diff:510-523,579-581). Before, it stayed running. If a later rocm serve (managed) reuses that daemon, a Ctrl-C on the foreground session removes health checks for that other service. The issue asks for daemon cleanup, so this isn't blocking; consider limiting it to the unexpected-exit path, or note the behaviour.
  • SIGTERM has a short unhandled window. The signal watcher is only registered at review.diff:747, after the child is spawned and the helper started (review.diff:581). A SIGTERM in that window still kills rocm with the default action.

Two unrelated hunks break existing e2e scenarios (serve-23 and dash-35), so the diff needs one more pass.

VERDICT: REVISE

mikeroysoft-agent and others added 8 commits October 24, 2026 11:05
Signed-off-by: mikeroysoft-agent <339795455+mikeroysoft-agent@users.noreply.github.com>
Signed-off-by: mikeroysoft-agent <339795455+mikeroysoft-agent@users.noreply.github.com>
Signed-off-by: mikeroysoft-agent <339795455+mikeroysoft-agent@users.noreply.github.com>
Signed-off-by: mikeroysoft-agent <339795455+mikeroysoft-agent@users.noreply.github.com>
Make the foreground scenario result helper const to satisfy the explicit
e2e clippy target. Regenerate the dependency manifest and third-party
notices with cargo-about 0.9.1 to match jsonc-parser 0.33.2 in Cargo.lock.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Signed-off-by: mikeroysoft-agent <339795455+mikeroysoft-agent@users.noreply.github.com>
Keep a disconnected signal receiver on Windows so foreground polling
uses the same instance interface without unused-self or const-fn lint
failures. Compile-check its no-stop behavior in a Windows-only test.
Correct the signal-hook cleanup comment.

Make the existing Windows transaction no-ops const and scope readonly
attribute lint expectations to Windows, where clearing the attribute
does not grant Unix permissions. These diagnostics blocked the same CI
job before its build and tests. Linux permissions are unchanged.

Reproduced all seven errors with the Windows GNU target before the fix.
Both workspace and explicit e2e Windows Clippy commands now pass, as do
the foreground SIGKILL and clean-stop lifecycle scenarios locally.
Native Windows execution and authenticated CI logs remain unavailable.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Signed-off-by: mikeroysoft-agent <339795455+mikeroysoft-agent@users.noreply.github.com>
…round mode

Signed-off-by: mikeroysoft-agent <339795455+mikeroysoft-agent@users.noreply.github.com>
Signed-off-by: mikeroysoft-agent <339795455+mikeroysoft-agent@users.noreply.github.com>
@mikeroysoft-agent

Copy link
Copy Markdown
Collaborator Author

{"type":"session","version":3,"id":"01a16f1f-2dd1-746f-8f01-7d1fe60dc26e","timestamp":"2026-10-24T18:14:36.753Z","cwd":"/tmp/factory-review-t1xpf3nc"}

Factory rejected reviewer evidence: review_exit:1.

@mikeroysoft-agent

Copy link
Copy Markdown
Collaborator Author

{"type":"session","version":3,"id":"01a16f26-84be-73b6-bd55-44a6ca42d948","timestamp":"2026-10-24T18:22:37.758Z","cwd":"/tmp/factory-review-7ymlpx7k"}

Factory rejected reviewer evidence: review_exit:1.

@mikeroysoft-agent

Copy link
Copy Markdown
Collaborator Author

{"type":"session","version":3,"id":"01a16f2c-cfaf-7723-bd89-011ef6970bf3","timestamp":"2026-10-24T18:29:30.159Z","cwd":"/tmp/factory-review-rjwqik3o"}

Factory rejected reviewer evidence: review_exit:1.

Signed-off-by: mikeroysoft-agent <339795455+mikeroysoft-agent@users.noreply.github.com>
@mikeroysoft-agent

Copy link
Copy Markdown
Collaborator Author

{"type":"session","version":3,"id":"01a16f35-9169-7497-8171-f85ccac9f24c","timestamp":"2026-10-24T18:39:04.041Z","cwd":"/tmp/factory-review-ewlb6bjc"}

Factory rejected reviewer evidence: review_exit:1.

Signed-off-by: mikeroysoft-agent <339795455+mikeroysoft-agent@users.noreply.github.com>
Signed-off-by: mikeroysoft-agent <339795455+mikeroysoft-agent@users.noreply.github.com>
@mikeroysoft-agent

Copy link
Copy Markdown
Collaborator Author

Verdict: REVISE. The diff fixes the root cause of the reported bug, but one change can stop a background daemon that other servers and automations depend on.

Spec: issue #73

The old code polled process_is_running(child_pid) (kill(pid,0), at crates/rocm-core/src/process.rs:371-383). That check returns true for an unreaped zombie. The supervisor child is a direct, unreaped child of rocm serve, so after it died, rocm serve waited forever. That matches the incident. The diff now keeps the Child and polls it with try_wait (review.diff:600-611, 704-716, 760-763), which fixes this.

Acceptance criterion Status Evidence
A test SIGKILLs the backend, sees a nonzero exit within a few seconds, and finds no orphaned processes Met review.diff:1075-1080, 1261-1288, 1360-1371, 1399-1411
A user SIGTERM/SIGINT to rocm serve exits 0 Met (the test covers SIGTERM; SIGINT takes the same path) review.diff:304-345, 686-688, 742-744, 1082-1086
The exit is logged with the backend's status or signal Met in production through the final log drain plus the engine-side messages review.diff:714, 761, 1059; existing Lemonade message at engines/lemonade/src/process.rs:589
Optional --restart-backend N Not done; it is optional —

The required documentation updates (README, --help, docs/testing.md, docs/manual-testing.md, per AGENTS.md §5) are present at review.diff:17-26, 46-66, 999-1046. The new docs link uses a relative path as required.

Required fixes

  1. Cleanup stops the shared background daemon even when other servers or automations still use it. Evidence: review.diff:416-429, 485-487, 527-545, 562-568.
    • What the daemon is: rocm daemon is a per-user singleton. The doc comment at apps/rocm/src/service_launch.rs:755-760 says it is "reused by both the serve --managed path and automations enable". The automations enable path (apps/rocm/src/automations.rs:45) and later rocm serve calls do not start a new daemon when one is already running (background_helper_already_running). The daemon has no idle exit.
    • Trigger: run rocm serve A --foreground while no daemon is running, so this invocation starts it. Then, from another shell, run rocm serve B (background) or rocm automations enable …; both reuse that daemon. Then press Ctrl-C in A, or let A's backend die.
    • Impact: stop_attached_service force-terminates the daemon. Server B loses health checks and auto-recovery, and the enabled automations stop. Nothing warns the user, and nothing restarts the daemon until some later command starts one. Before this diff, Ctrl-C left the daemon running. This removes supervision for other servers, which is the same kind of failure the issue is about.
    • Why this exceeds the spec: the criterion is "no orphaned … daemon". A daemon still supervising other servers or running automations is not orphaned.
    • Fix: before terminating spawned_daemon, skip it if any other managed service record is still running or any automation is enabled. This is one extra check, not a new abstraction. Never stopping the daemon would fail the "no orphaned daemon" criterion, which serve-27/28 test. With the check, the single-service case (the systemd unit, serve-27/28) still stops the daemon.

Optional suggestions

  • Windows cleanup can kill a process that reused the supervisor's PID. On Windows child is always None (review.diff:153-165). Cleanup captures ProcessIdentity::capture(child_pid) at cleanup time (review.diff:380, 395-398). Windows process_start_ticks always returns None (crates/rocm-core/src/proc_lifecycle.rs:318-320), so any live process holding that PID counts as a match. If the PID was reused after the supervisor exited (ServerExited(None)) or after StopServer succeeded, that process tree gets force-killed. Using the identity recorded at spawn time, or skipping the kill when child is None and the PID was already gone, would close this. It is not blocking because the window is short and Windows identity checks are best-effort elsewhere.
  • The SIGKILL log assertion only checks text the fixture writes itself. The signal: 9 (SIGKILL) line comes from the fixture's own echo (review.diff:181-182, 1393), not from Lemonade or vLLM. The CLI's own new line, "server process exited unexpectedly with status …" (review.diff:558, 569), is never asserted. AGENTS.md §3 asks that messages be asserted together with the behavior they describe; one contains("exited unexpectedly") check in serve-27 would cover it.
  • rocm services stop <id> from another shell now looks like a failure. That command stops the supervisor, so the attached serve reports "exited unexpectedly" and exits nonzero (review.diff:555-569). This matches the issue's "for any reason" wording. Checking the record's stop-requested state could separate this case from a real crash if that distinction matters later.
  • Signal handling adds a direct signal-hook dependency (review.diff:38). tokio with the signal feature is already a dependency of apps/rocm, and rocmd and rocm-dash-tui use tokio::signal. Since signal-hook is already in the lock graph through crossterm, no new crate enters the build. The net-new AttachSignalWatcher is justified by the clean-SIGTERM criterion: without a handler, SIGTERM kills the process before cleanup runs.
  • Useful comments were removed outside the change. The diff deletes rationale comments that still apply, notably the "enter raw mode before spawning the key reader" ordering note (review.diff:661-665) and the raw-mode \r\n note (review.diff:695-697).
  • One Windows test only checks a default. windows_signal_watcher_does_not_request_a_stop (review.diff:770-777) checks that a dropped sender reports no request, which is incidental behavior rather than a user-visible contract.

VERDICT: REVISE

Signed-off-by: mikeroysoft-agent <339795455+mikeroysoft-agent@users.noreply.github.com>
@mikeroysoft-agent

Copy link
Copy Markdown
Collaborator Author

Review of #73: rocm serve --foreground exit on backend death

The diff meets every required acceptance criterion in issue #73 and follows the repo's documented conventions. The optional --restart-backend N was not implemented, which the issue and brief allow (it's marked optional and off by default). I didn't run any builds or tests; the gate report is the authority for build and test status.

Root cause. Before this change, the attached path dropped the supervisor's Child without ever reaping it. Unix process_is_running uses kill(pid, 0), which still succeeds on an unreaped zombie (snapshot/crates/rocm-core/src/process.rs:378). So the old exit check could never see the dead supervisor, which matches the outage. The diff keeps the Child (review.diff:145, :202) and polls it with try_wait (review.diff:605-616). That fixes the cause, not just the symptom.

Spec (issue #73)

Criterion Status Evidence
Backend SIGKILL → nonzero exit within a few seconds, no orphaned backend or daemon Met review.diff:560-575 cleans up, then bail!s. Scenario serve-27 (review.diff:1146-1151) checks the exit is nonzero, takes under 5s, and that backend, supervisor and daemon identities are gone (review.diff:1376-1387, :1470-1482). Without the fix the sh supervisor stays a zombie, so this test would have failed.
SIGINT/SIGTERM to rocm serve → clean shutdown, exit 0 Met AttachSignalWatcher (review.diff:304-357) is checked before the child-exit check in both loops (review.diff:691, :747). The Stop path cleans up and returns Ok (review.diff:529-556). Covered by serve-28 (review.diff:1153-1157).
Exit logged with the backend's exit status/signal Met Lemonade already logs exited with status {status} (snapshot/engines/lemonade/src/process.rs:589). vLLM now logs it too (review.diff:1130). The log is drained one last time after the exit, so the final line reaches stdout and the journal (review.diff:719, :766). The parent also prints the supervisor's status (review.diff:563).
No orphaned daemon Met The daemon is stopped only if this invocation started it and nothing else needs it (review.diff:416-434). Unit tests cover the cases where it must survive (review.diff:950-1013).

New abstractions, each justified by a criterion:

  • Child is retained in ManagedSpawn: needed for reaping and the real exit status.
  • AttachSignalWatcher: needed for the SIGTERM → exit 0 criterion. It uses signal-hook 0.3.18, which was already in the tree through crossterm. Cargo.lock only gains an edge, and THIRD_PARTY_NOTICES.txt:5861 already lists the crate.
  • ensure_background_helper_running_quiet now returns the spawned daemon's identity: needed for the "no orphaned daemon" criterion.
  • drain_attached_log: needed for the final drain after exit, and it removes a duplicated loop.

Standards

  • AGENTS.md §3 (regression test plus Gherkin scenario for user-visible exit codes): met by serve-27 and serve-28 in tests/e2e-cucumber/features/model_serving.feature.
  • AGENTS.md §5 (update README, --help, docs/testing.md, docs/manual-testing.md together): done at review.diff:22-26, :52-54, :63-64, :1106-1117 and :1078-1094. The docs/testing.md link is relative, as required. No stale "Ctrl-C leaves the managed server running" claims remain in docs, skills or features.
  • e2e hook gating: the fixture is compiled in only with e2e-test-hooks on Unix. Otherwise foreground_backend_fixture_dir() is a const fn returning None (review.diff:115-126), so a production build ignores the environment variable. This matches the hook pattern in docs/release-trust.md.

Required fixes

None.

Optional suggestions (non-blocking)

  1. The serve-27 log assertion checks a line the fixture writes itself. The signal: 9 (SIGKILL) text comes from the fixture's own echo (review.diff:182), and the assertion is at review.diff:1464. That proves the final-drain path, not the parent's own message. Asserting the CLI's line server process exited unexpectedly with status … (review.diff:563) in the same scenario would fit AGENTS.md §3, "A message about the CLI's own behavior is asserted together with the behavior."
  2. Unrelated rationale comments were deleted. Examples: the launch-lock release reason (review.diff:451-453), raw mode entered before the key reader (review.diff:666-670), and the raw-mode \r\n explanation (review.diff:700-702; that code moved into drain_attached_log at review.diff:633-635 without its comment). The code they describe is unchanged and no repo rule requires keeping them, but restoring them keeps the diff focused on serve: exit nonzero when the backend server dies in --foreground mode #73.
  3. Short window before the signal watcher exists. A SIGTERM after the spawn (review.diff:460) but before AttachSignalWatcher::new() (review.diff:658) still uses the default action and leaves the detached supervisor running. Under systemd, the cgroup stop covers this. Registering the watcher before the spawn would close the gap, but nothing in the issue requires it.

VERDICT: APPROVE

@mikeroysoft-agent mikeroysoft-agent added factory-approved Reviewer APPROVE recorded by the factory; merge-stage precondition and removed factory-approved Reviewer APPROVE recorded by the factory; merge-stage precondition labels Oct 24, 2026
@mikeroysoft-agent

Copy link
Copy Markdown
Collaborator Author

Gate failed after rebase onto current main:

Gate report

Host toolchain

ROCm: unavailable
Kernel: 7.2.5-3-omarchy
amdgpu: unavailable

  • conflict-markers: PASS
  • clippy: PASS
  • tests: FAIL
  • smoke: PASS
  • leak-scan: PASS

tests failure

test therock::version_proptests::compare_version_strings_is_antisymmetric ... ok
test uninstall::tests::a_block_for_another_install_folder_stays ... ok
test uninstall::tests::a_block_in_a_profile_the_installer_created_leaves_it_empty ... ok
test therock::version_proptests::compare_version_strings_never_inverts_the_numeric_order ... ok
test uninstall::tests::a_folder_that_cannot_be_listed_is_an_error_not_an_empty_one ... ok
test uninstall::tests::a_program_whose_name_is_not_utf8_still_counts_as_another_program ... ok
test uninstall::tests::a_users_own_path_line_is_never_touched ... ok
test uninstall::tests::a_replacement_left_by_an_interrupted_uninstall_does_not_block_the_next ... ok
test uninstall::tests::an_unterminated_block_is_left_alone ... ok
test uninstall::tests::crlf_profiles_are_handled ... ok
test uninstall::tests::other_entries_ignore_rocm_cli_files_and_what_is_being_removed ... ok
test uninstall::tests::plan::a_block_written_through_a_linked_home_is_still_found ... ok
test uninstall::tests::plan::a_block_from_an_earlier_install_in_another_shell_is_also_undone ... ok
test uninstall::tests::plan::a_linked_profile_is_edited_where_it_lives_and_stays_a_link ... ok
test uninstall::tests::plan::a_failure_after_the_path_undo_still_reports_what_was_done ... ok
test uninstall::tests::plan::a_profile_that_is_not_utf8_is_still_edited_byte_for_byte ... ok
test uninstall::tests::plan::a_path_change_that_cannot_be_undone_stops_before_the_binaries_go ... ok
test uninstall::tests::plan::a_record_naming_a_profile_that_is_not_utf8_is_still_followed ... ok
test uninstall::tests::plan::a_recorded_profile_block_is_undone_and_the_record_goes_last ... ok
test uninstall::tests::plan::a_recorded_profile_that_cannot_be_read_stops_before_anything_goes ... ok
test uninstall::tests::plan::a_recorded_profile_without_the_block_is_reported_not_edited ... ok
test uninstall::tests::plan::a_recorded_spelling_is_matched_even_when_no_derived_one_is ... ok
test uninstall::tests::plan::an_install_folder_whose_path_is_not_utf8_is_still_found ... ok
test uninstall::tests::plan::an_unrecorded_profile_that_cannot_be_read_is_a_warning_not_an_absence ... ok
test uninstall::tests::plan::another_program_in_the_folder_keeps_the_change_and_its_record ... ok
test uninstall::tests::plan::keeping_the_binaries_keeps_the_path_change_and_says_so ... ok
test uninstall::tests::profile_exprs_cover_the_literal_and_home_relative_forms ... ok
test uninstall::tests::plan::without_a_record_the_marked_block_in_a_known_profile_is_found ... ok
test uninstall::tests::removing_the_block_restores_the_profile_as_it_was ... ok
test uninstall::tests::the_fish_block_and_the_literal_path_form_are_removed ... ok
test uninstall::tests::the_record_parses_each_kind_and_skips_blank_or_malformed_lines ... ok
test uninstall::tests::the_replacement_profile_is_never_more_readable_than_the_original ... ok
test uninstall::tests::the_review_lists_the_record_the_running_exe_and_the_path_changes ... ok
test uninstall::tests::the_running_executable_is_matched_whatever_case_windows_reports_it_in ... ok
test therock::archive_properties::sdk_tarball_unpack_extracts_a_benign_tarball_into_the_install_root ... ok
test therock::version_proptests::select_runtime_update_source_is_order_independent ... ok
test therock::version_proptests::select_startup_update_manifest_is_order_independent ... ok
test therock::version_proptests::select_runtimes_to_remove_is_order_independent ... ok
test therock::version_proptests::select_rocm_version_picks_the_newest_candidate ... ok
test therock::tests::wait_with_output_bounded_kills_a_slow_child_at_the_deadline ... ok
test remote::install::tests::resolvable_findings_on_their_own_do_promise_a_rerun ... ok
test therock::version_proptests::sorting_versions_is_order_independent ... ok
test therock::version_proptests::generator_reach_report ... ok
test advised_commands::every_advised_command_parses ... ok
test therock::version_proptests::select_rocm_version_is_order_independent ... ok
test tests::lemonade_stop_unloads_selected_model_over_http ... ok
test tests::services_prune_waits_for_the_managed_launch_lock_before_sweeping ... ok
test tests::serve_readiness_wait_withholds_ready_while_the_model_only_lists ... ok
test therock::archive_properties::sdk_tarball_unpack_never_writes_outside_the_install_root ... ok
test therock::tests::cli_family_overrides_a_stale_env_value_naming_an_unsupported_gpu ... ok
test therock::tests::env_override_is_ignored_end_to_end_without_the_opt_in ... ok
test therock::tests::extracting_the_sdk_archive_removes_it ... ok
test therock::tests::extracting_the_sdk_archive_reports_cleanup_failure ... ok
test therock::tests::next_layout_resolves_its_own_bases ... ok
test therock::tests::python_launcher_prefers_path_python_before_saved_managed_python ... ok
test therock::tests::python_launcher_prefers_path_python_over_managed_when_venv_capable ... ok
test therock::tests::render_update_json_installs_the_suppression_guard_around_resolution ... ok
test therock::tests::wait_with_output_bounded_waits_unbounded_when_no_timeout_is_given ... ok
test therock::tests::http_get_respects_max_time_secs ... ok
test therock::tests::metadata_signature_verification_accepts_generated_key_and_rejects_tamper ... ok
test comfyui::archive_properties::source_unpack_without_a_top_level_symlink_never_writes_outside_the_app_root ... ok
test comfyui::tests::reinstall_is_refused_while_comfyui_starts_until_it_is_stopped ... ok
test comfyui::tests::reinstall_is_refused_while_comfyui_runs_until_it_is_stopped ... ok
test tests::lemonade_stop_unload_is_bounded_by_the_request_timeout ... ok

failures:

---- comfyui::tests::status_reports_stopped_when_saved_comfyui_pid_is_gone stdout ----

thread 'comfyui::tests::status_reports_stopped_when_saved_comfyui_pid_is_gone' (3331758) panicked at apps/rocm/src/comfyui.rs:2758:9:
assertion failed: rendered.contains("status: stopped")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    comfyui::tests::status_reports_stopped_when_saved_comfyui_pid_is_gone

test result: FAILED. 1318 passed; 1 failed; 7 ignored; 0 measured; 0 filtered out; finished in 5.21s

error: test failed, to rerun pass `-p rocm --bin rocm`

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

serve: exit nonzero when the backend server dies in --foreground mode

2 participants