Skip to content

serve: exit nonzero when the backend server dies in --foreground mode #73

Closed

Description

@mikeroySoft

Problem
On ROCm tank, rocm serve ornith --foreground (user unit rocm-serve-ornith.service, Restart=on-failure) kept running after its backend died. On Mon 10/19 at about 6:34 PM PT the journal logged Error: Lemonade packaged llama-server exited with status signal: 9 (SIGKILL), but the rocm process (and its rocm daemon child) stayed up with nothing listening on port 11435. systemd saw the unit as active, never restarted it, and the factory's triage model was down for about 2.7 days.

Expected
In --foreground mode, when the backend server exits for any reason (signal or nonzero status), rocm serve exits nonzero after cleaning up its children, so the service manager's restart policy applies.

Acceptance criteria

  • A test (or scripted repro) kills the backend with SIGKILL and asserts rocm serve --foreground exits nonzero within a few seconds and leaves no orphaned backend or daemon process.
  • A clean backend shutdown still exits 0 when the user stopped it (SIGTERM/SIGINT to rocm serve).
  • The exit is logged with the backend's exit status/signal.
  • Optional: --restart-backend N to retry in-process before giving up (off by default).

Context: mikeroySoft/factory triage outage; companion factory tickets cover health checks and alerting.

Activity

  1. mikeroysoft-agent commented on Oct 22, 2026

    @mikeroysoft-agent
    Collaborator

    Triage: The issue has a clear problem statement (rocm serve --foreground keeps running after the backend dies, defeating systemd's Restart=on-failure), a precise expected behavior, and multiple observable acceptance criteria (nonzero exit on backend death, exit 0 on clean user-initiated shutdown, logging of backend exit status/signal, no orphaned processes). The optional --restart-backend item is explicitly marked optional. It is a contained bug fix within this repository that does not touch release, signing, or security policy, so it does not require human judgment.

    Agent brief: Make rocm serve --foreground exit nonzero when the backend server dies for any reason (signal or nonzero status), after cleaning up its child processes (backend + daemon), and log the backend's exit status/signal. It must still exit 0 on a clean user-initiated shutdown (SIGTERM/SIGINT sent to rocm serve). Add a test or scripted repro that SIGKILLs the backend and asserts rocm serve --foreground exits nonzero within a few seconds and leaves no orphaned backend or daemon process, plus a clean-shutdown case asserting exit 0. Optional (off by default): add --restart-backend N to retry the backend in-process before giving up. Verify with: start rocm serve <service> --foreground, then kill -9 <backend-pid> and assert the parent process exits nonzero and no orphan processes remain; separately send SIGTERM/SIGINT to rocm serve and assert exit 0.

  2. added
    ready-for-agentFully specified and ready for an AFK agent
    and removed
    needs-triageMaintainer needs to evaluate this issue
    on Oct 22, 2026
  3. added and removed
    ready-for-agentFully specified and ready for an AFK agent
    on Oct 22, 2026
  4. removed their assignment
    on Oct 22, 2026
  5. mikeroysoft-agent commented on Oct 22, 2026

    @mikeroysoft-agent
    Collaborator

    Factory dispatcher escalating: PR #74: CI failed (Third-party notices current, clippy); factory-approved label removed.

    Escalation packet: /home/mike/.local/share/district/roots/rocm-cli/.factory/escalations/73.md

    Worker handoff notes:

    Issue 73 handoff

    Changed

    • rocm serve --foreground now retains and reaps the managed supervisor child, detects its real exit status, logs an unexpected exit, cleans up the managed process tree and any daemon started by that invocation, and returns nonzero.
    • The vLLM supervisor now writes its backend child's ExitStatus before returning nonzero, so signal deaths such as SIGKILL are present in the engine log instead of being collapsed to exit code 1.
    • SIGINT/SIGTERM sent to an attached serve now follows the explicit user-stop path, performs the same cleanup, and returns zero. Ctrl-D detach behavior remains unchanged.
    • Added Linux/GPU-independent Cucumber scenarios serve-27 and serve-28. They use a real shell process tree to verify backend SIGKILL failure, signal logging, clean user SIGTERM, bounded exit time, and absence of backend/supervisor/daemon orphans.
    • Updated rocm serve help, README, testing docs, and manual-testing instructions.
    • Fixed gate blockers encountered while completing the required full gate: PATH serialization for TheRock archive tests, a stale-port ComfyUI test race, non-UTF-8 system-tar diagnostics escaping Rust test capture, and an ETXTBSY race in the system SDK fake enumerator.
    • Eight signed-off commits are on agent/73; .factory-prompt.md remains untracked and uncommitted as required.

    Verified

    • cargo xtask e2e -- -n 'serve-(27|28)': 2 scenarios, 7 steps passed.
    • cargo test -p rocm-engine-vllm: 108 tests passed across 3 suites.
    • cargo test -p rocm-core uv_archive_extraction_never_writes_outside_the_staging_dir: passed; captured output is valid UTF-8.
    • cargo test -p rocm-core --lib system_sdk::tests::probe_collects_deduplicated_gfx_targets_dropping_gfx000: passed.
    • /home/mike/.local/share/uv/tools/factory/bin/python -m factory gate --report .factory/gate-report-73.md: PASS for conflict markers, clippy, tests, smoke, and leak scan.

    Still unverified

    • The host has no ROCm/amdgpu runtime, so live Lemonade and vLLM backend SIGKILLs on GPU hardware were not exercised. The committed E2E fixture covers the same parent/supervisor/backend/daemon lifecycle with real Unix processes.

    Next

    • Dispatcher may push agent/73 and open the PR. CI or a ROCm host should run the live manual SIGKILL reproduction if hardware validation is required.
  6. mikeroysoft-agent commented on Oct 22, 2026

    @mikeroysoft-agent
    Collaborator

    Factory dispatcher escalating: PR #74: CI failed (Test (affected crates), build-and-test, windows-build-and-test, Coverage (rocm-dash crates, ratcheted)); factory-approved label removed.

    Escalation packet: /home/mike/.local/share/district/roots/rocm-cli/.factory/escalations/73.md

    Worker handoff notes:

    Issue 73 handoff

    Changed

    • rocm serve --foreground now retains and reaps the managed supervisor child, detects its real exit status, logs an unexpected exit, cleans up the managed process tree and any daemon started by that invocation, and returns nonzero.
    • The vLLM supervisor now writes its backend child's ExitStatus before returning nonzero, so signal deaths such as SIGKILL are present in the engine log instead of being collapsed to exit code 1.
    • SIGINT/SIGTERM sent to an attached serve now follows the explicit user-stop path, performs the same cleanup, and returns zero. Ctrl-D detach behavior remains unchanged.
    • Added Linux/GPU-independent Cucumber scenarios serve-27 and serve-28. They use a real shell process tree to verify backend SIGKILL failure, signal logging, clean user SIGTERM, bounded exit time, and absence of backend/supervisor/daemon orphans.
    • Updated rocm serve help, README, testing docs, and manual-testing instructions.
    • Fixed gate blockers encountered while completing the required full gate: PATH serialization for TheRock archive tests, a stale-port ComfyUI test race, non-UTF-8 system-tar diagnostics escaping Rust test capture, and an ETXTBSY race in the system SDK fake enumerator.
    • Eight signed-off commits are on agent/73; .factory-prompt.md remains untracked and uncommitted as required.

    Verified

    • cargo xtask e2e -- -n 'serve-(27|28)': 2 scenarios, 7 steps passed.
    • cargo test -p rocm-engine-vllm: 108 tests passed across 3 suites.
    • cargo test -p rocm-core uv_archive_extraction_never_writes_outside_the_staging_dir: passed; captured output is valid UTF-8.
    • cargo test -p rocm-core --lib system_sdk::tests::probe_collects_deduplicated_gfx_targets_dropping_gfx000: passed.
    • /home/mike/.local/share/uv/tools/factory/bin/python -m factory gate --report .factory/gate-report-73.md: PASS for conflict markers, clippy, tests, smoke, and leak scan.

    Still unverified

    • The host has no ROCm/amdgpu runtime, so live Lemonade and vLLM backend SIGKILLs on GPU hardware were not exercised. The committed E2E fixture covers the same parent/supervisor/backend/daemon lifecycle with real Unix processes.

    Next

    • Dispatcher may push agent/73 and open the PR. CI or a ROCm host should run the live manual SIGKILL reproduction if hardware validation is required.
  7. mikeroysoft-agent commented on Oct 22, 2026

    @mikeroysoft-agent
    Collaborator

    Factory handoff request 73/2: a human decision is needed on #73.

    No decision owner could be determined (unassigned); a maintainer should claim it.

    Question
    How should #73 proceed? It escalated with: PR #74: CI failed (Test (affected crates), build-and-test, windows-build-and-test, Coverage (rocm-dash crates, ratcheted)); factory-approved label removed. Automatic recovery stopped because the manager's 1 allowed round(s) are exhausted.

    Evidence (bounded links; worker logs and the escalation packet stay on the runner)

    Proposed next step
    Read the evidence, then amend the ticket and relabel ready-for-agent for another attempt, fix the branch by hand, request changes on the PR, or close it. Replying here is context for humans; nothing here retries, approves or merges.

    Routing

    • decision_owner: absent — ticket body has no **Decision owner** section
    • initiative: skipped — initiative owner only routes requirements, not ci
    • configuration: absent — no [collaboration] section in .factory.toml; legacy behaviour unchanged
      To claim or reassign this decision, set a **Decision owner** section (a GitHub login) in the issue body; the next pass reflects it and never overwrites it.
  8. mikeroysoft-agent commented on Oct 23, 2026

    @mikeroysoft-agent
    Collaborator

    Factory dispatcher escalating: review requested changes after manager FIX.

    Escalation packet: /home/mike/.local/share/district/roots/rocm-cli/.factory/escalations/73.md

    Worker logs: /home/mike/.local/share/district/roots/rocm-cli/.factory/logs/73-attempt-6.log

    Worker handoff notes:

    Issue 73 handoff

    Changed

    • rocm serve --foreground now retains and reaps the managed supervisor child, detects its real exit status, logs an unexpected exit, cleans up the managed process tree and any daemon started by that invocation, and returns nonzero.
    • The vLLM supervisor now writes its backend child's ExitStatus before returning nonzero, so signal deaths such as SIGKILL are present in the engine log instead of being collapsed to exit code 1.
    • SIGINT/SIGTERM sent to an attached serve now follows the explicit user-stop path, performs the same cleanup, and returns zero. Ctrl-D detach behavior remains unchanged.
    • Added Linux/GPU-independent Cucumber scenarios serve-27 and serve-28. They use a real shell process tree to verify backend SIGKILL failure, signal logging, clean user SIGTERM, bounded exit time, and absence of backend/supervisor/daemon orphans.
    • Updated rocm serve help, README, testing docs, and manual-testing instructions.
    • Fixed gate blockers encountered while completing the required full gate: PATH serialization for TheRock archive tests, a stale-port ComfyUI test race, non-UTF-8 system-tar diagnostics escaping Rust test capture, and an ETXTBSY race in the system SDK fake enumerator.
    • Eight signed-off commits are on agent/73; .factory-prompt.md remains untracked and uncommitted as required.

    Verified

    • cargo xtask e2e -- -n 'serve-(27|28)': 2 scenarios, 7 steps passed.
    • cargo test -p rocm-engine-vllm: 108 tests passed across 3 suites.
    • cargo test -p rocm-core uv_archive_extraction_never_writes_outside_the_staging_dir: passed; captured output is valid UTF-8.
    • cargo test -p rocm-core --lib system_sdk::tests::probe_collects_deduplicated_gfx_targets_dropping_gfx000: passed.
    • /home/mike/.local/share/uv/tools/factory/bin/python -m factory gate --report .factory/gate-report-73.md: PASS for conflict markers, clippy, tests, smoke, and leak scan.

    Still unverified

    • The host has no ROCm/amdgpu runtime, so live Lemonade and vLLM backend SIGKILLs on GPU hardware were not exercised. The committed E2E fixture covers the same parent/supervisor/backend/daemon lifecycle with real Unix processes.

    Next

    • Dispatcher may push agent/73 and open the PR. CI or a ROCm host should run the live manual SIGKILL reproduction if hardware validation is required.
  9. mikeroysoft-agent commented on Oct 23, 2026

    @mikeroysoft-agent
    Collaborator

    Factory dispatcher escalating: review requested changes after manager FIX.

    Escalation packet: /home/mike/.local/share/district/roots/rocm-cli/.factory/escalations/73.md

    Worker logs: /home/mike/.local/share/district/roots/rocm-cli/.factory/logs/73-attempt-7.log

    Worker handoff notes:

    Issue 73 handoff

    Changed

    • rocm serve --foreground now retains and reaps the managed supervisor child, detects its real exit status, logs an unexpected exit, cleans up the managed process tree and any daemon started by that invocation, and returns nonzero.
    • The vLLM supervisor now writes its backend child's ExitStatus before returning nonzero, so signal deaths such as SIGKILL are present in the engine log instead of being collapsed to exit code 1.
    • SIGINT/SIGTERM sent to an attached serve now follows the explicit user-stop path, performs the same cleanup, and returns zero. Ctrl-D detach behavior remains unchanged.
    • Added Linux/GPU-independent Cucumber scenarios serve-27 and serve-28. They use a real shell process tree to verify backend SIGKILL failure, signal logging, clean user SIGTERM, bounded exit time, and absence of backend/supervisor/daemon orphans.
    • Updated rocm serve help, README, testing docs, and manual-testing instructions.
    • Fixed gate blockers encountered while completing the required full gate: PATH serialization for TheRock archive tests, a stale-port ComfyUI test race, non-UTF-8 system-tar diagnostics escaping Rust test capture, and an ETXTBSY race in the system SDK fake enumerator.
    • Eight signed-off commits are on agent/73; .factory-prompt.md remains untracked and uncommitted as required.

    Verified

    • cargo xtask e2e -- -n 'serve-(27|28)': 2 scenarios, 7 steps passed.
    • cargo test -p rocm-engine-vllm: 108 tests passed across 3 suites.
    • cargo test -p rocm-core uv_archive_extraction_never_writes_outside_the_staging_dir: passed; captured output is valid UTF-8.
    • cargo test -p rocm-core --lib system_sdk::tests::probe_collects_deduplicated_gfx_targets_dropping_gfx000: passed.
    • /home/mike/.local/share/uv/tools/factory/bin/python -m factory gate --report .factory/gate-report-73.md: PASS for conflict markers, clippy, tests, smoke, and leak scan.

    Still unverified

    • The host has no ROCm/amdgpu runtime, so live Lemonade and vLLM backend SIGKILLs on GPU hardware were not exercised. The committed E2E fixture covers the same parent/supervisor/backend/daemon lifecycle with real Unix processes.

    Next

    • Dispatcher may push agent/73 and open the PR. CI or a ROCm host should run the live manual SIGKILL reproduction if hardware validation is required.
  10. mikeroysoft-agent commented on Oct 24, 2026

    @mikeroysoft-agent
    Collaborator

    Factory handoff request 73/4: a human decision is needed on #73.

    No decision owner could be determined (unassigned); a maintainer should claim it.

    Question
    How should #73 proceed? It escalated with: review requested changes after manager FIX. Automatic recovery stopped because the manager's 3 allowed round(s) are exhausted.

    Evidence (bounded links; worker logs and the escalation packet stay on the runner)

    Proposed next step
    Read the evidence, then amend the ticket and relabel ready-for-agent for another attempt, fix the branch by hand, request changes on the PR, or close it. Replying here is context for humans; nothing here retries, approves or merges.

    Routing

    • decision_owner: absent — ticket body has no **Decision owner** section
    • initiative: skipped — initiative owner only routes requirements, not implementation
    • configuration: absent — no [collaboration] section in .factory.toml; legacy behaviour unchanged
      To claim or reassign this decision, set a **Decision owner** section (a GitHub login) in the issue body; the next pass reflects it and never overwrites it.
  11. 1 remaining item

  12. added and removed
    ready-for-agentFully specified and ready for an AFK agent
    on Oct 24, 2026
  13. removed their assignment
    on Oct 24, 2026
  14. mikeroysoft-agent commented on Oct 24, 2026

    @mikeroysoft-agent
    Collaborator

    Factory dispatcher escalating: REVISE verdict after 3 review round(s).

    Escalation packet: /home/mike/.local/share/district/roots/rocm-cli/.factory/escalations/73.md

    Worker logs: /home/mike/.local/share/district/roots/rocm-cli/.factory/logs/73-attempt-6.log

    Worker handoff notes:

    Issue 73 handoff

    Changed

    • rocm serve --foreground retains and reaps its managed supervisor, treats an unexpected backend/supervisor exit as an error, logs the exit status or signal, and cleans up its managed process tree and daemon.
    • SIGINT/SIGTERM sent to an attached serve follows the explicit user-stop path, cleans up, and exits zero; Ctrl-D detach remains unchanged.
    • The vLLM supervisor logs its backend child's ExitStatus before returning an error.
    • Linux/GPU-independent Cucumber scenarios serve-27 and serve-28 exercise real backend/supervisor/daemon processes for SIGKILL failure and clean SIGTERM shutdown.
    • Updated rocm serve help, README, testing documentation, and manual verification instructions.
    • Preserved the current-base serve-23 and dash-35 coverage while removing stale-base changes unrelated to issue 73.
    • Made the ComfyUI stale-state test deterministic after the required full gate exposed its released-port race: port 0 is always unreachable and tests the intended stopped-state branch without racing another listener.
    • Nine signed-off commits are on agent/73; .factory-prompt.md, the gate report, and this handoff remain uncommitted.

    Verified

    • cargo xtask e2e -- -n 'serve-(23|27|28)|dash-35': 4 scenarios, 15 steps passed.
    • cargo test -p rocm --bin rocm comfyui::tests::status_reports_stopped_when_saved_comfyui_pid_is_gone: passed.
    • /home/mike/.local/share/uv/tools/factory/bin/python -m factory gate --report .factory/gate-report-73.md: PASS for conflict markers, clippy, tests, smoke, and leak scan.

    Still unverified

    • This host has no ROCm/amdgpu runtime, so live Lemonade and vLLM backend SIGKILL behavior was not exercised on GPU hardware.
    • serve-27 proves the final log drain and process lifecycle with a real Unix process tree, but its fixture emits the signal: 9 (SIGKILL) line itself; it does not independently validate Lemonade's production signal-formatting text.
    • PR agent/73: serve: exit nonzero when the backend server dies in --foreground mode #74's latest three reviewer runs exited with review_exit:1 and published no code finding; no source change can address that reviewer-process failure.

    Next

  15. mikeroysoft-agent commented on Oct 24, 2026

    @mikeroysoft-agent
    Collaborator

    Factory handoff request 73/5: a human decision is needed on #73.

    No decision owner could be determined (unassigned); a maintainer should claim it.

    Question
    How should #73 proceed? It escalated with: REVISE verdict after 3 review round(s). Automatic recovery stopped because no manager is configured, so nothing recovers automatically.

    Evidence (bounded links; worker logs and the escalation packet stay on the runner)

    Proposed next step
    Read the evidence, then amend the ticket and relabel ready-for-agent for another attempt, fix the branch by hand, request changes on the PR, or close it. Replying here is context for humans; nothing here retries, approves or merges.

    Routing

    • decision_owner: absent — ticket body has no **Decision owner** section
    • initiative: skipped — initiative owner only routes requirements, not implementation
    • configuration: absent — no [collaboration] section in .factory.toml; legacy behaviour unchanged
      To claim or reassign this decision, set a **Decision owner** section (a GitHub login) in the issue body; the next pass reflects it and never overwrites it.
  16. added
    ready-for-agentFully specified and ready for an AFK agent
    and removed on Oct 24, 2026
  17. added and removed
    ready-for-agentFully specified and ready for an AFK agent
    on Oct 24, 2026
  18. removed their assignment
    on Oct 24, 2026
  19. mikeroysoft-agent commented on Oct 24, 2026

    @mikeroysoft-agent
    Collaborator

    Factory dispatcher escalating: PR #74: gate failed after rebase onto moved main; factory-approved label removed.

    Escalation packet: /home/mike/.local/share/district/roots/rocm-cli/.factory/escalations/73.md

    Worker handoff notes:

    Issue 73 handoff

    Changed

    • rocm serve --foreground retains and reaps its managed supervisor, treats an unexpected backend/supervisor exit as an error, logs the exit status or signal, and cleans up its managed process tree and daemon.
    • SIGINT/SIGTERM sent to an attached serve follows the explicit user-stop path, cleans up, and exits zero; Ctrl-D detach remains unchanged.
    • The vLLM supervisor logs its backend child's ExitStatus before returning an error.
    • Linux/GPU-independent Cucumber scenarios serve-27 and serve-28 exercise real backend/supervisor/daemon processes for SIGKILL failure and clean SIGTERM shutdown.
    • Foreground cleanup now preserves a daemon started by that invocation when another managed service is still running or automations are enabled. Two Linux process-lifecycle regression tests cover both shared-daemon cases.
    • Updated rocm serve help, README, testing documentation, and manual verification instructions.
    • Twelve signed-off commits are on agent/73; .factory-prompt.md, the gate report, and this handoff remain uncommitted.

    Verified

    • Regression tests failed before the shared-daemon fix: both cleanup cases terminated the daemon.
    • cargo test -p rocm --bin rocm cleanup_preserves_its_daemon -- --nocapture: 2 passed.
    • cargo test -p rocm --bin rocm service_launch::tests::attached_lifecycle: 9 passed.
    • cargo fmt --all -- --check: passed.
    • cargo xtask e2e -- -n 'serve-(27|28)': 2 scenarios, 7 steps passed.
    • /home/mike/.local/share/uv/tools/factory/bin/python -m factory gate --report .factory/gate-report-73.md: PASS for conflict markers, clippy, tests, smoke, and leak scan.

    Still unverified

    • The gate reports ROCm and amdgpu unavailable, so live Lemonade and vLLM backend SIGKILL behavior was not exercised on GPU hardware.
    • serve-27 proves the final log drain and process lifecycle with a real Unix process tree, but its fixture emits the signal: 9 (SIGKILL) line itself; it does not independently validate Lemonade's production signal-formatting text.

    Next

  20. mikeroysoft-agent commented on Oct 24, 2026

    @mikeroysoft-agent
    Collaborator

    Factory handoff request 73/6: a human decision is needed on #73.

    No decision owner could be determined (unassigned); a maintainer should claim it.

    Question
    How should #73 proceed? It escalated with: PR #74: gate failed after rebase onto moved main; factory-approved label removed. Automatic recovery stopped because the manager's 3 allowed round(s) are exhausted.

    Evidence (bounded links; worker logs and the escalation packet stay on the runner)

    Proposed next step
    Read the evidence, then amend the ticket and relabel ready-for-agent for another attempt, fix the branch by hand, request changes on the PR, or close it. Replying here is context for humans; nothing here retries, approves or merges.

    Routing

    • decision_owner: absent — ticket body has no **Decision owner** section
    • initiative: skipped — initiative owner only routes requirements, not implementation
    • configuration: absent — no [collaboration] section in .factory.toml; legacy behaviour unchanged
      To claim or reassign this decision, set a **Decision owner** section (a GitHub login) in the issue body; the next pass reflects it and never overwrites it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions