Skip to content

ROCMAI-456: let a surviving dashboard take over the telemetry daemon - #1040

Open
rominf wants to merge 9 commits into
mainfrom
ROCMAI-456-fix-dash-daemon-handoff
Open

rominf wants to merge 9 commits into
mainfrom
ROCMAI-456-fix-dash-daemon-handoff

Conversation

@rominf

@rominf rominf commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Why

On Linux/WSL, with two rocm dash sessions open, quitting (or killing) the one that started first leaves the other showing disconnected until it is restarted.

The embedded telemetry daemon ran inside whichever dashboard started first, and whether a dashboard hosts it was decided once, at startup. When the host exited, no surviving dashboard ever took over. Removing or guarding the exit-time socket cleanup alone does not fix this: with the first dashboard killed by SIGKILL the socket file stays behind and the second session is still stranded.

A second defect sat next to it: the daemon unlinked whatever was at the socket path before binding, so a second daemon at the same path replaced a live listener (reproduced: a new inode at the path, and the old daemon and its clients unreachable).

Once several dashboards contend for one path, what a daemon may do with the socket's directory matters on every attempt, so the directory policy is settled here too rather than left as it was.

What changes

  • Every live dashboard runs a keeper (apps/rocm/src/dash_daemon.rs, new) that serves the socket whenever no other daemon does. It retries every 250 ms while another daemon of this build owns the path, every 5 s while something without the ownership lock answers there (an older build, or another program), and backs off from 1 s to 30 s on any other failure. The backoff and the failed-attempt count start over when another daemon is found serving, when the socket is taken from a serving daemon, or after a run that served for at least 30 s; a run that served for less does not reset them, so a daemon that keeps stopping soon after it starts is still backed off rather than restarted every second. When the owning dashboard closes or is killed, a surviving one takes over and its TUI reconnects, typically within 1–2 s.
  • The socket's directory is checked before anything is created, opened, probed or removed in it (check_socket_dir and the pure socket_dir_verdict in crates/rocm-dash-daemon/src/server.rs), judged by what it resolves to when it is a symlink:
    • a missing directory is created at mode 0700, as is every missing parent the recursive create makes on the way to it, so none of them is chmodded afterwards; it is then checked again like an existing one, since another user could have created it in between;
    • something that is not a directory is refused, first;
    • a sticky directory (such as /tmp) is never chmodded, whoever owns it, root included; it is used when root or this user owns it, with a warning once the path is claimed when others may write in it, and refused when another user owns it;
    • any other directory owned by another user is refused, before anything is touched in it;
    • a non-sticky directory this user owns is tightened to 0700 whenever group or others have any access to it (mode & 0o077 != 0), so an owned 0755 one is tightened too; the chmod is logged once, at info, with the directory and its old mode. One that group and others cannot reach at all (0700, 0500) is left exactly as it is, so a dashboard polling for the socket never repeats a chmod or the log line.
    • this applies to the directory of whatever unix: socket path the dashboard serves, including one configured in dashboard.tui.connect: when this user owns that directory and it is not sticky, it is restricted to 0700, so such a path belongs in a dedicated directory, not $HOME or one shared with others.
  • Socket ownership is an exclusive lock on <socket>.lock (mode 0600, opened with O_NOFOLLOW, so a symlink planted at the lock path is refused rather than followed) next to the socket, held for the daemon's life and released by the OS however the process exits (SocketClaim). A daemon that finds the path owned returns the public SocketInUse error (saying whether a lock holder or an unlocked listener holds it) and removes or replaces nothing at the path.
  • Holding the lock, a daemon removes an existing socket only when it is stale. The probe connects and waits for the greeting (reading at most 4 KiB of it) before hanging up; a connection that is refused means stale, while an answer, a timeout or a full backlog means something is serving and the path is left alone, and any other connect error (one the socket's mode causes, for example) fails the claim with that error and leaves the path alone too. A path that is not a socket, symlinks included, is never removed: the daemon refuses with an error naming the path. A bind that loses a race to something that appeared at the path is classified by what the path names then: the not-a-socket refusal for something that is not a socket, and SocketInUse otherwise, including a path that vanished or cannot be inspected by then. On exit the daemon removes the socket only if the path is still the one it bound.
  • A serving daemon watches its path. Every 2 s it checks that the path still names its socket; if the socket was removed or replaced it stops with the public SocketReplaced error and releases the lock, so a keeper can claim the path again without counting a failure. A check that cannot read the path is tolerated; five unreadable checks in a row give the path up.
  • The daemon owns its tasks. The runner and the per-client tasks run in JoinSets owned by the serving future, so stopping the daemon stops them and closes its clients' connections (the runner's own background GPU detection task is not among them). A runner that stops or panics ends the daemon with an error, and the keeper restarts it with backoff; a panicking client task is logged.
  • A daemon that was failing when the dashboard closed is reported after it closes. The keeper records the failure its daemon is stuck in, and rocm dash prints it to stderr once the TUI has left the terminal. The report opens with "could not start" when the daemon never served and "stopped serving" when it had served and has not served again; it names the path and the last error, adds "(N failed attempts)" from two failed runs on, and names the client log directory only when the warnings actually reached the log. It is dropped when another daemon answers at the socket by the time the dashboard closes, since that dashboard was then working on the other daemon. A panic of the keeper itself is always reported, as "the dashboard's supervision of it stopped unexpectedly" with the panic's own message (and the failure recorded before it, if there was one and it was not dropped). While the dashboard waits about 5 s at most for its daemon to start (it can run over by one probe of a listener that does not greet, up to 0.5 s), it probes the socket the same way the stale-socket check does, reading the greeting before it hangs up, so the daemon does not log a client error for every dashboard opened; it stops waiting as soon as the daemon has failed.
  • The dashboard rebuilds its benchmark rows on (re)connect. A daemon that took over re-reads the benchmark CSV and replays it, so the TUI clears its benchmark list when it connects and takes the replay as the whole list. Daemon and TUI share one cap for that list (BENCH_HISTORY_CAP in rocm-dash-core), so clearing loses nothing.
  • The e2e harness's PTY-spawned rocm drops the caller's RUST_LOG (piped runs still inherit it), so steps that count the info-level lines in the client log do not depend on the log filter of the shell running the suite. This is test-harness hardening; the CLI's own behaviour does not change.
  • The embedded daemon's eprintln! on exit, which wrote into the TUI's alternate screen, is replaced with tracing.
  • New dependencies: libc for rocm-dash-daemon on unix only (geteuid and O_NOFOLLOW; already in the lockfile through rocm-core), and tokio's test-util feature as a rocm dev-dependency, for the keeper's paused-clock timing tests.

User-visible changes:

  • A dashboard left open after the owning one closes or is killed reconnects instead of staying disconnected, and keeps its chart history.
  • After a hand-off each benchmark row is shown once: the list is rebuilt from what the new daemon re-reads from the benchmark CSV. The exception is a row the daemon records in the moment a dashboard subscribes, which can reach that dashboard both in the replay and in the live stream and so appear twice around a reconnect; that race is pre-existing (the subscribe replay and the broadcast are unchanged from main) and is not fixed here.
  • A dashboard opened after a hand-off gets only what the new daemon has recorded since it took over (its snapshot ring holds the last 30 snapshots, about 30 s).
  • GPU figures can stay empty for up to about 15 s after a hand-off while the new daemon detects the GPU through amd-smi again (its 5 s detection limit plus its 10 s limit on the first system query).
  • Each hand-off starts a new session-*.ndjson recording, since the daemon that takes over is a new daemon.
  • A <socket>.lock file exists beside the socket and stays after the last dashboard closes.
  • A regular file or symlink at the socket path is left in place, where it used to be deleted.
  • After the dashboard closes, a report of why its telemetry daemon could not start or stopped serving is printed to stderr, as described above. This also holds when the dashboard is opened from interactive rocm chat or the bare rocm launcher; from the launcher the lines land on the main screen and are seen when the launcher itself exits.
  • Socket-directory behaviour compared with main, which created the directory and then always chmodded it to 0700, refusing it only when that chmod failed and it was not a root-owned sticky directory: a non-sticky directory this user owns that group or others have any access to, such as a 0755 one, is still tightened to 0700 as on main, and the change is now logged at info; one they cannot reach at all keeps its mode (a 0500 one is no longer made 0700); a sticky directory is no longer chmodded by anyone, so root no longer chmods /tmp when its socket lives there, and a sticky directory of this user's own keeps its mode; and a directory another user owns (other than a root-owned sticky one such as /tmp) is refused without a chmod being attempted, so a daemon running as root now refuses it where main chmodded it and used it (as a regular user main already refused it, after its chmod failed).
  • rocm dash --help and the README now say that open dashboards share one telemetry daemon on Linux/WSL, that a surviving one takes it over, and when the failure report is printed, including that a crash of the dashboard's own supervision of its telemetry daemon is reported even when another dashboard's daemon is serving. The README also no longer says the dashboard auto-starts an embedded rocmd: the embedded daemon is the dashboard's own telemetry daemon. Its build section no longer calls rocmd the dashboard's telemetry daemon either, and says what rocmd does run: the watchers, the managed-service supervisor and the MCP tools.
  • rocm chat --help now says that on Linux/WSL the interactive chat shares the dashboard's telemetry daemon the way rocm dash does and, like it, prints why that daemon was failing after it closes.

Windows is unaffected: there is no embedded daemon or socket there, so no keeper starts and nothing is reported.

Docs updated in the same change: README.md, docs/architecture.md (the new file entry, the socket-directory policy and the socket-ownership boundary), docs/testing.md, and docs/manual-testing.md (section 4e, Two Dashboards At Once, including the hand-off costs, the failure report, how to see the "stopped serving" report by hand, and how to see a configured socket path's directory restricted to 0700 and the one log line it leaves).

Decisions for the reviewer

  • std::fs::File::try_lock instead of rocm_core::FileLock. FileLock cannot create the lock file with mode 0600, and using it would add a production rocm-dash-daemon → rocm-core dependency edge for one call.
  • The directory check runs before the claim and changes nothing in an acceptable directory. A dashboard that does not own the daemon polls four times a second, so the check must be idempotent: the only chmod is on a non-sticky directory of ours that group or others have any access to, which is then 0700 on the next attempt and is neither chmodded nor logged again. The shared-sticky-directory warning is given once, by the daemon that claims the path, not on every poll.
  • A sticky directory is never chmodded. The sticky bit already means only an entry's owner may remove or rename it, so the 0600 socket and lock cannot be replaced there by another user; chmodding a shared directory such as /tmp (which root could do) would break it for everyone else.
  • "Served long enough" is 30 s, the longest backoff wait. Serving at all is not enough (a daemon serves as soon as it binds), nor is the first snapshot (a runner that sends one and fails is the same loop). A run that lasted as long as the longest wait restarts at most about as often as the backoff would allow anyway.
  • A failure is not reported when another daemon serves at close, but a keeper panic always is. The failure is recorded before the keeper waits to retry, so another dashboard's daemon may have taken the socket in that wait; a panic is why nothing retries any more, whatever else is serving.
  • Public API additions in rocm-dash-daemon: SocketInUse.holder with the SocketHolder enum, SocketReplaced, the NOT_A_SOCKET constant, run_with_ready (a callback once the daemon is serving) and socket_answers (the claim's liveness probe). The keeper is the only in-repo caller; run keeps its signature.
  • A runner that stops or panics ends the whole daemon, and the keeper restarts it with backoff. The alternative, a daemon that keeps accepting clients with no runner behind it, looks connected and shows nothing. This trades possible restart churn for a failure that is visible in the log and in the exit report.
  • Clear-on-reconnect is applied to benchmark rows only. Chart history is kept, so a surviving dashboard does not lose its charts at a hand-off.

Scope

  • Out of scope: hardening where the socket is published (the default socket directory and its fallbacks). In particular, in a shared sticky directory another user can pre-create the <socket>.lock file and hold its lock (or leave it unopenable), which keeps every dashboard from serving at that path. The fix is not to publish the socket in a shared directory at all, which belongs with that work.
  • Only the socket's own directory is checked; its ancestors are not. A bare socket file name such as unix:rocmdashd.sock is created in the current directory, which is not checked at all.
  • The race in which another user creates the missing directory between the check and its creation is covered only through the test seam that runs the check as a different uid; a real second user is not exercised.
  • Pre-existing, not changed here: a dashboard that reconnects to the same daemon gets that daemon's up to 30 most recent snapshots replayed and appends them, so a few recent chart points can appear twice. It will be tracked separately; the README and docs/manual-testing.md describe it as a known issue.
  • Mixed versions: a dashboard from an older build takes no lock, so it can still replace a socket served by this build. Close old dashboards when upgrading.
  • Follow-up, to be tracked separately: when the lock holder never answers at the socket (for example, a home directory shared across hosts, where the holder runs on another machine), the keeper keeps waiting for it without reporting anything.
  • Known limitation: the "is the socket still ours" check (on exit and in the path watch) compares (dev, ino). If an older build without the lock, or a manual bind, recreates the path after an external unlink and the filesystem reuses the inode number, the daemon could take that socket for its own. Daemons from this build cannot produce that sequence, because a second one cannot bind while the lock is held.

Verification

New tests, each seen to fail with the code it covers broken:

  • crates/rocm-dash-daemon/src/server.rs, by area:
    • socket directory: the policy for every owner and mode as a pure function, including sticky directories, owned by root or by this user, where only group or only others may write, each treated as shared (the_socket_directory_policy); a missing directory created private, every directory the recursive create makes on the way included, with no chmod logged (creates_missing_parent_at_0700, serving_creates_a_missing_socket_directory_privately); only a directory of ours that group or others have any access to tightened, before the claim, an owned 0755 one included and logged once with its old mode (only_a_socket_directory_others_can_reach_is_tightened, tightens_owned_preexisting_parent, serving_restricts_a_socket_directory_it_owns, a_loose_socket_directory_of_ours_is_made_private_before_the_claim, a_loose_socket_directory_is_tightened_before_the_claim_runs); another user's directory refused before anything is touched, never chmodded, also through a symlink (a_socket_directory_another_user_owns_is_refused_before_anything_is_touched, another_users_writable_socket_directory_is_refused_not_chmodded, a_symlink_to_another_users_socket_directory_is_refused) and when the daemon had just created it (a_socket_directory_just_created_is_judged_like_an_existing_one); a sticky directory never chmodded and reported as shared, with the warning given once by the daemon that claims the path (a_sticky_socket_directory_is_served_as_it_is, preexisting_root_sticky_parent_is_used_as_it_is, a_sticky_socket_directory_is_reported_as_shared, the_shared_directory_warning_comes_once_from_the_daemon_that_claims_the_path); a directory that cannot be inspected or is not a directory (a_socket_directory_that_cannot_be_inspected_is_reported, a_socket_directory_that_is_not_a_directory_is_refused); a symlinked directory judged by its target (a_symlinked_socket_directory_is_judged_by_its_target); a symlink at the lock path not followed (a_symlink_at_the_lock_path_is_not_followed); the helper that gives a bare socket file name no directory to check, rather than the empty path the check refuses (a_bare_socket_name_has_no_directory_to_check; it calls the helper only, since serving a bare name in a unit test would change the process-wide current directory under parallel tests). The cross-user cases run the check as a different uid through a test seam (ServeSeams::our_uid), since a test cannot create a directory another user really owns.
    • ownership: a live socket is refused rather than replaced; a lock holder keeps the path even when its socket looks stale; a stale socket from a dead daemon is replaced; a path that is not a socket is never removed (asserting both the error naming the path and that the file is still there); dropping a claim removes only its own socket; exactly one of several concurrent daemons binds; an answering socket with no lock holder is left in place.
    • probe and bind: a busy or slow listener counts as answering, not stale; the probe waits for the greeting before hanging up, and reads at most a bounded amount of it from a listener that never ends a line (the_probe_reads_a_bounded_greeting_from_a_listener_that_never_ends_a_line); a socket the probe cannot connect to for a reason other than refusal, here its mode, is reported rather than replaced (a_socket_the_probe_cannot_reach_is_reported_not_replaced); a bind lost to a listener without the lock is reported in use; a file that appears before the bind is refused as not a socket; what a bind that finds the path taken reports for a socket, a file, and a path that vanished since (a_bind_that_finds_the_path_taken_reports_what_is_there_now); and a chmod that fails after binding still leaves the claim to remove its socket (a_claim_whose_chmod_fails_still_removes_its_socket).
    • path watch: a replaced socket gives the path up; a briefly unreadable path keeps serving; one that stays unreadable gives it up; exactly five unreadable checks in a row give it up, with the fifth check's error, and a readable check starts the count over (only_five_unreadable_path_checks_in_a_row_give_the_path_up).
    • task lifecycle: a stopped daemon closes its clients' connections, also while its broadcast channel lives; its runner stops with it; a daemon stops when its runner does; finished client tasks are released.
  • apps/rocm/src/dash_daemon.rs (keeper): a surviving dashboard takes over when the owner leaves; closing a dashboard that never owned the daemon leaves it running; the pure retry decision next_step (backoff doubling to 30 s, resets on an owner, on a reclaim and after a run that served long enough, 5 s polling for an unlocked listener, counting only failures in a row); a daemon refused its path is retried and reported after the dashboard closes (the report names the path once and the log, the log holds the warning, the file is untouched); the report names the log only when the warning reached it, and counts failed attempts from two; a failure ends, and is not reported, once another daemon serves the socket; a daemon that recovered is not reported; a failed daemon does not hold up the dashboard; started against a real daemon, the dashboard stops waiting as soon as the real probe finds the socket answering, well inside its limit, with nothing to report on close (the_dashboard_waits_only_until_its_daemon_answers); the up-to-5 s wait before the dashboard draws, as its own function on tokio's paused clock, ends at the first probe that finds the socket answering (the_ready_wait_ends_once_the_socket_answers), at once when the daemon has already failed (the_ready_wait_ends_at_once_when_the_daemon_has_failed), and after its limit otherwise (the_ready_wait_gives_up_after_its_limit); a removed socket is served again without a failure; a keeper that panicked is reported, with its message, as having stopped when it had served, ahead of a failure recorded before it, and even when another daemon serves; a non-unix target starts no keeper. The keeper's timing is checked with the daemon scripted on tokio's paused clock: failed runs spaced by the growing backoff (failed_runs_are_spaced_by_the_growing_backoff), a daemon that keeps stopping soon after it starts serving backed off rather than restarted every second (a_daemon_that_keeps_stopping_soon_after_serving_backs_off), the backoff starting over after a run that served long enough (a_run_that_served_long_enough_starts_the_backoff_over), a daemon that served and then stopped reported as having stopped (a_daemon_that_served_and_then_stopped_is_reported_as_stopped), a replaced socket claimed again one poll later without a failure (a_replaced_socket_is_claimed_again_after_one_poll_without_a_failure), and a socket in use tried again after one 250 ms takeover poll when another daemon holds its lock and after 5 s when something without the lock answers there, neither recorded as a failure (a_socket_in_use_is_tried_again_after_its_holders_poll).
  • crates/rocm-dash-tui/src/app/mod.rs: benchmark_rows_are_held_once_across_a_reconnect.

Scenarios in tests/e2e-cucumber/features/dash.feature. Each simulates a GPU-less host itself (like dash-37), so the runner's GPU does not change the outcome. dash-55 to dash-57 are tagged only @requires-os:linux, so they run on every Linux e2e lane, GPU or not, and are skipped on Windows; dash-58 also carries @requires-dir-permissions, since a directory permission does not bind root, and is skipped where the suite runs as root.

  • dash-55, @id:dash-second-session-survives-first-closing: opens two dashboards, waits until both connections are in the client log, quits the first, and requires the second to connect again and show the connected header. Against the pre-fix code it fails at "the second dashboard is connected to its telemetry daemon again".
  • dash-56, @id:dash-reports-unusable-telemetry-socket-on-exit: puts a regular file where the socket goes, opens and quits the dashboard, and requires it to exit successfully, report after closing that its telemetry daemon could not use that file and where the warning was logged (and the log to hold the warning), and leave the file in place.
  • dash-57, @id:dash-handoff-lists-each-benchmark-result-once: records two benchmark results, opens two dashboards (waiting, like dash-55, for both connections in the client log before the first is quit), and requires the second to list each result once on its Observe view, both before and after the first is quit and it has connected to the daemon that took over. This also drives the event loop's call to on_connected.
  • dash-58, @id:dash-reports-telemetry-daemon-that-stopped-serving: makes the socket's directory unreadable under a serving daemon, waits until the daemon gives the socket up, requires the dashboard to show it is disconnected and, after it closes, to report that its telemetry daemon stopped serving (naming the socket), not that it could not start.

Run locally (Linux, no WSL), on the final tree, rebased on main at 9e50eac: cargo fmt --all -- --check; cargo clippy --locked --workspace --all-targets -- -D warnings; cargo clippy --locked -p e2e-cucumber --test e2e -- -D warnings; cargo xtask check-module-size, check-architecture-doc and check-crate-edges; cargo test for rocm-dash-daemon, rocm-dash-tui and rocm-dash-core (unit and integration tests), and the rocm binary's dash tests (run from the built test binary); and cargo xtask e2e -- -n "dash-5[5-8]", where dash-55 to dash-58 passed, 35 of 35 steps. All passed. The tests that capture log lines through a thread-local tracing subscriber, in rocm-dash-daemon and in the rocm binary's keeper tests, install it through a log_at test helper in each crate that first registers two process-lifetime no-op dispatchers: tracing-core caches a callsite's interest when it is first hit, and while at most one dispatcher is registered it asks only the hitting thread's default subscriber, so a test running in parallel with no subscriber could cache a callsite as disabled and the capturing test then saw no line. With the helper, the rocm-dash-daemon unit tests ran 60 times, ten processes at once, with 0 failures; with that registration disabled, the same loop failed 4 of 40 runs, each with tightens_owned_preexisting_parent seeing no log line. The stale-socket test fixture likewise waits, within the test deadline, until a connection to the dropped listener is refused, because a child process a parallel test starts can hold the listener's descriptor until it execs. The four tests named above for the probe, the bind, the chmod and a bare socket name were each seen to fail with the branch they cover broken: the probe's other errors taken as stale or as answering, the bind's not-a-socket check inverted or applied to a vanished path, the socket's identity recorded after the chmod instead of before, and the helper passing a bare name's empty parent to the directory check.

Not verified locally:

  • The Windows build: no cross toolchain on the host, so the Windows CI job is authoritative. The keeper tests are unix-gated, and on Windows stop_keeper and failure_report are unreachable at runtime because spawn_keeper starts nothing there.
  • The two mode-000 path-watch tests and a_socket_the_probe_cannot_reach_is_reported_not_replaced print a skip line and count as passed when run as root.
  • cargo xtask coverage: not run locally.
  • The license-header (hawkeye) and lychee hooks: not installed on the host; CI runs both.
  • cargo xtask smoke: not run on the final tree. On this host it stops at its rocmd sandbox examine snapshot step, unrelated to this change: bubblewrap cannot configure the sandbox's loopback interface under the host's user-namespace policy.
  • The full cargo test --workspace --all-targets was not run on the final tree (only the dash crates and tests above), nor the dash scenarios other than dash-55 to dash-58; the CI e2e lanes run the whole feature. An earlier full run on this branch had failures that were all environmental (a noexec /tmp, git signing inside scratch repositories) and reproduced identically on main.

Checklist

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows.
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md (the clap declaration in cli.rs and the dispatch wiring in main.rs/lib.rs are expected, not a violation).
  • If this PR adds, renames or removes a production source file (or a .rs file under xtask/src or tests/e2e-cucumber), docs/architecture.md has the matching entry (cargo xtask check-architecture-doc fails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify.
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will not happen.
  • Comments and doc lines this PR adds or changes that state how the code behaves were checked against that code, and each that is the reason for a decision names the file and the function, constant or test that makes it true (AGENTS.md §3). For example, the socket-directory comments name socket_dir_verdict, the info-level chmod log names DEFAULT_FILTER in apps/rocm/src/logging.rs as what keeps it, and the ready-wait comment names the test proving the probe reads the greeting.
  • Any existing scenario this PR moves off the blocking mock e2e job (for example by a tag) is justified as a deletion (AGENTS.md §8). Not applicable: no existing scenario is moved or retagged; dash-55 to dash-58 are new.
  • Commits answering review feedback were diffed whole against the reviewed head and widen no documented user-facing option's meaning or authority (AGENTS.md §11). Not applicable: this PR has received no review.
  • Every commit message on the branch describes what its own commit does in words still true at the branch head (AGENTS.md §5).

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

…ROCMAI-456)

On Linux the telemetry daemon behind `rocm dash` ran inside whichever
dashboard started first. Nothing took it over when that process exited,
so every other dashboard still open stayed on "disconnected" until it
was restarted. The daemon also unlinked any existing socket before
binding, so a second daemon at the same path could orphan a live
listener and every client attached to it.

Each live dashboard now runs a keeper that serves the socket whenever no
other daemon does. Ownership is an exclusive lock on a file next to the
socket, held for the daemon's life and released by the OS however the
process exits; a daemon that finds the path owned returns SocketInUse
and the keeper retries, so a surviving dashboard takes over when the
owner closes or is killed. Holding the lock, a daemon removes an
existing socket only when a connection to it is refused, never removes
a path that is not a socket, and on exit removes the socket only if it
is still the one it bound.

Covered by unit tests for each state of the socket path and for the
keeper's takeover, and by scenario dash-55, which opens two dashboards,
quits the first and requires the second to reconnect.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…CMAI-456)

The daemon now owns its runner and client tasks and stops them when it
stops, closing its clients' connections. A runner that stops or panics
ends the daemon with an error, so the keeper restarts it.

A serving daemon checks every 2 s that the socket path is still its
own: if the socket was removed or replaced it gives the path up
(SocketReplaced); an unreadable path is tolerated, and five unreadable
checks in a row give it up. The stale-socket probe waits for the
greeting before hanging up, a listener that holds no lock is polled
every 5 s instead of every 250 ms, and a bind that loses to something
at the path reports it as not a socket when what is there now is not
one, and as in use otherwise.

The keeper counts only failures in a row and starts its backoff over
once another daemon is found or the socket was taken from a serving
daemon. It records the failure its daemon is stuck in,
and rocm dash prints it after the dashboard closes, naming the client
log only when the warning was actually logged.

A dashboard starts its benchmark list over on each connection, from
the rows the daemon replays, since a daemon that took over re-reads
the whole benchmark CSV and would otherwise send rows the dashboard
already lists; daemon and dashboard share BENCH_HISTORY_CAP, so the
replay holds as many rows as the dashboard keeps.

Adds scenario dash-56 for the failure report and updates the README,
architecture, testing and manual-testing docs.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The socket directory is now checked before the claim creates, opens or
probes anything in it. Another user's non-sticky directory is refused.
A sticky directory is never chmodded and is used when root or this
user owns it. A non-sticky directory this user owns is restricted to
0700 before it is used when group or others can reach it (a later
commit settles exactly which modes count). A directory the daemon had
to create is checked again afterwards, and the lock file is opened
with O_NOFOLLOW.

A run that served starts the keeper's backoff and failure count over
only once it has served for 30 s, so a runner that keeps stopping soon
after it starts is still backed off.

The report after the dashboard closes says "could not start" or
"stopped serving" by whether the daemon had served, and is dropped when
another daemon serves the socket by then. A keeper panic is always
reported, with its message.

Adds unit tests, including keeper timing on a paused clock, and
scenarios dash-57 (benchmark rows listed once after a hand-off) and
dash-58 (a daemon that stopped serving is reported as such), and
updates the help text, README and docs.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…-456)

The dashboard's wait for its daemon to start now probes with
socket_answers, which reads the daemon's greeting before hanging up, so
the daemon no longer logs a client error for every dashboard opened.

The stale-socket probe reads at most 4 KiB of greeting, so a listener
that streams bytes without ending a line is still answered promptly
rather than buffered until the probe's timeout.

The helper that counts unreadable socket-path checks now takes the
check's result and returns the error that gives the path up, so the
serving loop has no unreachable branch.

The PTY-spawned rocm in the e2e harness drops the caller's RUST_LOG, so
steps that count the client log's info-level lines do not depend on
the filter in the shell running the suite.

User-facing text no longer says "keeper": a crash of the dashboard's
own supervision of its telemetry daemon is reported as such in the
failure report, help text, README and manual-testing guide.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…I-456)

A non-sticky socket directory this user owns is now restricted to 0700
whenever group or others have any access to it (mode & 0o077), as main
did, rather than only when they may write in it, so an owned 0755
directory is tightened again. Sticky directories such as /tmp are still
never chmodded, and another user's non-sticky directory is still
refused. The tightening is logged once, at info, with the directory and
its old mode; a directory already private is left alone and logs nothing.

Test the keeper's wait while another daemon owns the socket on a paused
clock: one takeover poll when the lock holder serves it, the slower poll
when something without the lock answers there, and neither recorded as a
failure. Update the architecture, testing and manual-testing docs to the
new rule, including that a configured dashboard.tui.connect socket path
has its directory restricted too.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Pull the empty-parent filter for a bare socket name such as
`unix:rocmdashd.sock` out into a pure function and test it: such a path
has no directory to check, and the check would otherwise refuse the empty
path, so the daemon would never serve. The test drives the helper only;
serving a bare name would change the process-wide current directory
under parallel tests. Say in the comments why that directory is left
unchecked, and note the exception where the daemon's directory policy
is documented.

Add tests for more of the claim's branches: a probe that fails for a
reason other than refused (the socket's mode forbids connecting), which
is reported rather than replaced as stale; a bind that loses the path,
reported by what the path names now (a socket, a file, or a path that
vanished since); and a chmod that fails after binding, after which the
claim still removes the socket it bound. List them in docs/testing.md.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the ROCMAI-456-fix-dash-daemon-handoff branch from 3a4e6e8 to 26b4c85 Compare November 17, 2026 13:14
…c (ROCMAI-456)

The tests that capture the dash daemon's and the keeper's log lines
failed intermittently when run in parallel. tracing-core caches each
callsite's interest the first time the callsite is hit, and while at
most one dispatcher is registered it asks only the default subscriber
of the thread that hit it. A parallel test with no subscriber could
therefore cache a callsite as never enabled, and the capturing
subscriber then missed the line. Each test's log_at helper now first
registers two process-lifetime no-op dispatchers, so tracing-core asks
every live dispatcher and caches the same interest on every thread.

The stale-socket fixture now waits, within the test deadline, until a
connection is refused instead of asserting it on the first try. A child
process started by a parallel test can briefly inherit the listener's
descriptor until it execs, so the socket can still accept just after
the listener is dropped.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…CMAI-456)

The up-to-5 s wait for the dashboard's daemon socket to answer before
the dashboard draws is now a separate function, so a test on tokio's
paused clock can drive each of its three exits: the socket answers, the
limit is reached, and the daemon has already failed.

README no longer calls rocmd the dashboard's telemetry daemon: that
daemon runs inside `rocm dash`, and rocmd runs the watchers, the
managed-service supervisor and the MCP tools.

The e2e steps reuse the client-log reader and share one whitespace
squash helper instead of repeating both inline.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
… real probe (ROCMAI-456)

The socket directory test now checks that every directory the recursive
create makes, intermediate ones included, is created at 0700 rather than
chmodded afterwards, and the directory policy table gains sticky rows
where only group or only others may write. spawn_keeper is tested
against a real daemon, so the dashboard's ready-wait is seen to end on
the socket answering rather than on its time limit.

dash-55/57 now wait for both dashboards' connection lines before taking
the baseline count, since the client log is written through a
non-blocking appender and a late line could pass for the reconnection.

The chat help says the interactive chat shares the dashboard's daemon
and reports its failure after closing; the dashboard wording names
Linux/WSL; doc comments and docs/architecture.md are corrected to match
the code (keeper retry after one poll, the bind-loss refusal, the
ready-wait's possible overrun).

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf marked this pull request as ready for review November 17, 2026 15:27
@rominf
rominf requested a review from a team as a code owner November 17, 2026 15:27
@rominf
rominf requested a review from siloteemu November 17, 2026 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant