Skip to content

fix(dash): restore terminal on SIGTERM/SIGINT and make typed Ctrl-C quit - #326

Merged
r0x0r merged 14 commits into
mainfrom
fix/dash-restore-terminal-on-sigterm
Sep 16, 2026
Merged

r0x0r merged 14 commits into
mainfrom
fix/dash-restore-terminal-on-sigterm

Conversation

@r0x0r

@r0x0r r0x0r commented Aug 28, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

A signal delivered to rocm dash — kill <pid> (SIGTERM), a supervisor stop,
or an externally delivered kill -INT — took the default disposition and killed
the process while it still held the terminal in raw mode on the alternate
screen. The best-effort teardown after the event loop never ran, leaving the
shell in a broken display state (no cursor, raw mode on, stuck in the
alt-screen) that needs reset.

Bare rocm was worse. It is a persistent hub: it outlives each session's Tokio
runtime and returns to a synchronous, raw-mode launcher menu between flows.
Tokio never unregisters the libc handler it installs, so once a session's
runtime is dropped nothing drains the signal pipe — the menu sat in raw mode,
catching and discarding SIGTERM/SIGINT (unkillable) after the user's first flow.

And a typed Ctrl-C did nothing at all, on either platform. enable_raw_mode
clears ISIG (Unix) and ENABLE_PROCESSED_INPUT (Windows), so while the TUI is
up the terminal driver does not translate that keystroke into a SIGINT or a
CTRL_C_EVENT — the process receives the byte 0x03 as an ordinary key event,
and neither key loop handled it. Measured on Linux through the PTY driver: the
dashboard and the launcher both ignored 0x03 indefinitely (no exit within 15 s,
still on the alternate screen, cursor still hidden); only a SIGTERM got them out.
The same clearing of ENABLE_PROCESSED_INPUT is why tokio::signal::windows:: ctrl_c() cannot fire while the TUI is up; CTRL_BREAK_EVENT is unaffected, so
the Ctrl-Break arm is the one that genuinely works there.

Fix

  • app::run installs a termination-signal watcher (spawn_termination_watcher)
    before entering raw/alternate-screen mode. On SIGTERM/SIGINT (Unix) or
    Ctrl-Break (Windows) it restores the terminal and exits 128 + signo
    (SIGINT → 130, SIGTERM → 143). Registration is propagated via ? before any
    terminal state is mutated. The task is aborted on the clean-return path.
    Scope stated narrowly, because it is easy to overclaim: this covers signals
    that arrive as signals, plus anything delivered in the startup window before
    raw mode. Windows ctrl_c() is still registered, but only that startup window
    and a cross-process GenerateConsoleCtrlEvent can reach it.
  • Typed Ctrl-C is handled as a key, in both of the process's key loops — the
    dashboard event loop and the launcher's synchronous menu — and routed through
    the same restore_terminal + exit-130 path a real SIGINT takes, so the
    gesture cannot mean one thing in a session and nothing at the front door. It
    sits above every overlay so no screen can trap it. The single exception is a
    console for a still-running job, where Ctrl+C already means "cancel this
    job" (ui::job_console::on_console_key) — the documented way to stop a focused
    install/serve without truncating it mid-write; that meaning wins there. The
    exception ends when the job does: a finished console is dismissible but no
    longer cancellable, so exempting it would make the keystroke a silent no-op.
    It is now documented on both ? help modals, the launcher hint line and the
    README, alongside q.
  • dash::run_launcher keeps one multi_thread(1) runtime alive for the
    whole hub and spawns a single process-lifetime watcher on it, so every window
    — the launcher menu and each session — has a live reactor to restore the
    terminal and exit, across the session runtimes built and dropped in the loop.
  • Two watchers, one shutdown. When bare rocm escalates from the hub into a
    dashboard session, the hub's process-lifetime watcher and app::run's session
    watcher coexist, and Tokio's signal registry is process-global — one kill
    wakes both. A process-global SHUTTING_DOWN: AtomicBool arbitrates inside the
    watcher body (claim_shutdown / await_termination), so only the first
    watcher through restores the terminal and exits; the other stands down. Two
    threads therefore never race unsynchronised write_restore_sequences writes to
    stdout, nor call std::process::exit concurrently. The latch is process-global
    rather than a flag threaded through ResolvedArgs so it is path-independent:
    every call site, present and future, inherits it without having to know whether
    an outer watcher already exists. The typed-Ctrl-C path claims the same latch,
    so it cannot race a watcher either.
  • Watcher vs renderer, now closed. A draw can land after
    restore_terminal() returns: the watcher runs on a Tokio worker thread while
    frames are drawn on the thread that owns the loop (block_on's thread for a
    session, the main thread for the synchronous hub menu), and nothing locks the
    terminal. It cannot re-enter the alternate screen — EnterAlternateScreen is
    emitted exactly once at startup and Terminal::draw never re-emits it — but
    every frame ends by hiding or repositioning the cursor and repainting, so a
    late frame would undo the show-cursor half of the restore and paint a stale
    dashboard over the restored screen. Rather than a process-global lock on every
    terminal write, both render loops now check the same SHUTTING_DOWN latch
    before drawing (app::shutdown_claimed); the latch is claimed before the
    restore begins, so no frame can start after the claim. What remains is a frame
    already in flight when the claim lands, which can interleave during the
    restore — out-of-order escapes on a terminal being reset in the same breath,
    with the restore bytes written last. That one is cosmetic and is accepted, and
    the reasoning is recorded on restore_terminal rather than left implicit.
  • Teardown is factored into restore_terminal; the escape-sequence half is
    write_restore_sequences<W: Write> so it is unit-testable against an
    in-memory sink without mutating the shared terminal. One implementation for
    the whole crate
    : run_launcher's open-coded disable_raw_mode +
    LeaveAlternateScreen + show_cursor was the second one, and now calls
    restore_terminal too — which also stops a vanished controlling terminal
    turning a clean launcher exit into an Err via those ?s.
  • run's clean-quit teardown claims the same latch before restoring. abort()
    alone is not enough: it cannot stop a watcher that has already resumed past its
    .await and is inside its own synchronous restore_terminal(); process::exit(code). A SIGTERM landing in the same instant the user presses
    q would otherwise have two threads writing restore sequences to one stdout,
    unsynchronised. This was the one restore path that did not claim the latch.

process::exit on a signal (or a typed Ctrl-C) is deliberate (see
spawn_termination_watcher docs): we stop promptly rather than race an orderly
teardown, so an in-flight focused-install child is left to the OS (matching the
default disposition) and the embedded daemon socket self-heals on the next bind.

Signal handling uses tokio::signal (already available via tokio's full
feature) — no new runtime dependency. libc is added as a Unix-only
dev-dependency so the unit tests can raise(3) a real signal in-process.

Tests

Unit tests (in-process, so the coverage gate sees them):

  • termination_watcher_parks_until_aborted — drives the real wiring:
    spawn_termination_watcher() registers successfully, the spawned task stays
    parked on recv() while no signal has arrived, and abort() cancels it (the
    clean-return path app::run uses).
  • termination_signals_yield_shell_exit_codes_and_only_one_watcher_shuts_down
    (Unix) — registers two TerminationSignals (the hub + session shape),
    raises one real SIGTERM (then one SIGINT), and asserts the first watcher
    receives it and maps it to 143/130 while the second stands down with
    None. Safe to self-signal because it drives await_termination directly —
    the watcher body that calls process::exit is never run — and because the
    handlers are installed before the raise. Serialised against the test above by
    a module-local lock, because the required Windows lane runs cargo test
    (threads in one process) rather than cargo nextest.
  • only_the_first_caller_claims_the_shutdown_latch — claim_shutdown returns
    true exactly once, including under 16-thread contention.
  • claiming_the_shutdown_latch_suspends_rendering — claim_shutdown /
    shutdown_claimed_on against a bare local latch. This is the latch predicate
    only; it deliberately does not claim to cover the render gate (see below).
  • a_claimed_shutdown_stops_the_dashboard_painting_another_frame and
    a_claimed_shutdown_stops_the_launcher_painting_another_frame — the render
    gate itself, in each of the two key loops. The gate was extracted out of
    the loop bodies into draw_frame_unless_shutting_down /
    draw_menu_unless_shutting_down, parameterised over the backend and the latch,
    precisely so it is reachable from a test: each drives a TestBackend, and
    asserts painted cells (not the predicate) — non-empty with an open latch,
    empty with a claimed one.
  • ctrl_c_is_recognised_as_a_key_because_raw_mode_suppresses_the_signal —
    Ctrl+C matches; a bare c, Ctrl+D and Ctrl+Shift+C (commonly bound to copy)
    do not.
  • ctrl_c_exits_the_session_but_a_job_console_keeps_cancelling_the_job — the
    precedence above: terminate with no console up, fall through to the console's
    cancel while a job is running.
  • ctrl_c_exits_once_the_console_job_has_finished — the other half of that
    precedence, and a real bug this PR shipped before review caught it: nothing
    clears a manager's active_job when its job completes (only an Esc/Enter
    dismissal does), so the console stays on screen. Keying the exemption on "a
    console is displayed" rather than "a job is running" therefore made Ctrl-C over
    a finished console fall through to on_console_key → CancelJob, which
    the reducer drops on a terminal job — the keystroke did nothing at all, in
    exactly the gesture this PR exists to fix.
  • write_restore_sequences_leaves_alt_screen_and_shows_cursor — asserts the
    emitted restore bytes against an in-memory sink.

Every one of the above was falsified: breaking claim_shutdown, swapping the
SIGTERM/SIGINT exit-code mapping, making the watcher body return instead of
parking, stripping LeaveAlternateScreen from write_restore_sequences,
dropping the modifier check from is_ctrl_c, dropping the job-console exception
from ctrl_c_should_exit, reverting the terminal-job check in
ctrl_c_should_exit to the old !has_active_console(), and deleting the
shutdown_claimed_on early-return from both render gates — each turns exactly
the covering test(s) red.

Correction. An earlier revision of this section claimed the last of those —
"making the render gate ignore the latch" — was already falsified. It was not:
at that point the gate was inlined in the two loop bodies and no test reached it,
so deleting it left the suite fully green (736 passed, 0 failed). @rominf caught
this and was right. The gate has since been extracted into the two testable
helpers above and is now genuinely covered; the falsification quoted here was
re-run against the current code, not asserted from memory.

Two paths remain uncovered by construction, stated plainly rather than papered
over: the claim_shutdown guard now wrapping run's clean-quit teardown, and
the register-before-raw-mode ordering. Both live in run, which enables raw mode
and takes over a real terminal; neither is reachable from a unit test, and the
race each closes is a sub-millisecond window a 20 ms-polled PTY scenario cannot
observe deterministically.

The two await_termination awaits in
termination_signals_yield_shell_exit_codes_and_only_one_watcher_shuts_down are
now wrapped in a 10 s tokio::time::timeout. A regression that breaks
registration of one signal kind (rather than mis-mapping its exit code) used to
park the .await forever and burn the lane's job timeout reporting nothing;
measured with such a regression injected, the un-timed test ran 119.76 s without
reporting, and the bounded one fails in 10.00 s with a named message.

A note recorded on SIGNAL_TEST_LOCK that the lock cannot fix: registering the
listeners installs Tokio's libc handler process-wide and Tokio never unregisters
it, so from the first of these tests onward a single-process cargo test run
(the required Windows lane) is deaf to SIGINT/SIGTERM. Harmless as the suite
stands, but it is global state and future tests must not assume the default
disposition is still in place.

Linux PTY scenarios in dash.feature (all @requires-os:linux, no GPU, run on
the Linux E2E lanes):

  • @id:dash-sigterm-restores-terminal (dash-12) — steady-state SIGTERM to a
    running dashboard → 143 + terminal restored.
  • @id:dash-sigint-restores-terminal (dash-13) — an externally delivered
    SIGINT → 130 + terminal restored. Explicitly not the Ctrl-C gesture; the
    step shells out to kill(1).
  • @id:dash-launcher-sigterm-restores-terminal-across-a-session (dash-14) —
    the launcher round-trip: open bare rocm, escalate into the dashboard,
    quit back to the menu (dropping that session's runtime), then SIGTERM → 143 +
    terminal restored. This is the across-session hub path a single-session
    scenario cannot reach. Verified red without the hub watcher, green with it.
    The process under test is the hub, so its outcome steps are the launcher's
    (the launcher exits from the signal with code …), not the dashboard's.
  • @id:dash-ctrl-c-restores-terminal (dash-15) — types a literal 0x03 at the
    dashboard through the PTY → 130 + terminal restored. The gesture a user
    actually performs, and the one nothing covered.
  • @id:dash-launcher-ctrl-c-restores-terminal (dash-16) — the same keystroke
    at the hub's front door, whose key loop is a separate one → 130 + terminal
    restored.

dash-15/dash-16 were falsified end to end: with the two key-loop handlers
removed, both fail (the process never exits and the wait times out); with them
in place, dash-12 … dash-16 all pass.

The startup register-before-raw-mode ordering is covered by construction
(spawn_termination_watcher runs before enable_raw_mode); a 20 ms-polled PTY
scenario cannot observe that microsecond window, so none is claimed for it.

Not verified locally. The Windows console behaviour above (raw mode
suppressing CTRL_C_EVENT, Ctrl-Break unaffected) is reasoned from the
documented ENABLE_PROCESSED_INPUT semantics and mirrors the measured Unix
result; the 0x03 experiment and the new scenarios were run on Linux only, and
dash-12 … dash-16 are @requires-os:linux. The Windows lanes exercise the
unit tests and the build, not the keystroke.

Harness timing change beyond this PR's scope

TuiSession::wait_for_exit_code now drains the PTY's final frame before
returning, because the terminal-restore sequences arrive immediately before the
process exits and restoration is asserted after it. That drain is not gated
on the caller: it also runs for wait_for_exit and therefore quit_and_wait,
i.e. for every existing PTY scenario that quits normally, none of which this PR
otherwise touches. It ends as soon as the reader thread sees EOF — normally one
20 ms poll — with DRAIN_TIMEOUT (250 ms) as the ceiling a wedged PTY could
impose. Flagged because it is a timing change to callers outside this PR's
intent, not because any of them regressed: the full dash-* set (16 scenarios,
100 steps) passes.

@r0x0r
r0x0r requested a review from a team as a code owner August 28, 2026 12:39
@r0x0r
r0x0r requested a review from siloteemu August 28, 2026 12:39

@r0x0r r0x0r left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tackling the broken-terminal-on-signal case — the bounded, best-effort restore_terminal() helper, the 128 + signo exit codes, and the signal_task.abort() on a clean return are all correct and nicely tested.

One substantive interaction worth deciding explicitly: the signal task ends in std::process::exit(code), which skips stack unwinding and therefore the kill_on_drop teardown of any in-flight focused install/serve child. On a clean quit, event_loop returning unwinds the runtime and kills those children; on the signal path they are now left running (orphaned). This isn't a regression versus today's default disposition (which also skips kill_on_drop), but this PR is the natural place to decide the intended behavior: should a SIGTERM'd rocm dash leave a mid-write focused install running, or terminate it with the child? The comment at focused_close_key_blocked shows the truncation hazard is already on your radar, so a one-line statement of intent in the code or PR body would close the question either way.

Minor related note (not blocking): once tokio's signal handlers are installed they stay registered process-wide even after signal_task.abort(), so during the brief clean-exit teardown window a SIGTERM is caught-and-ignored rather than terminating. The process is exiting anyway, so this is fine to leave — just worth being aware of.

@michaelroy-amd michaelroy-amd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #326 review — request changes

Head reviewed: 4048d69d8fcfd1b70f2a1a157cccdfbc28e5afdc

Standards

Blocking — Windows build is red and Ctrl-Break is not handled

crates/rocm-dash-tui/src/app/mod.rs:1688-1699

The PR claims Windows Ctrl-C and Ctrl-Break support, but tokio::signal::ctrl_c() subscribes only to CTRL_C_EVENT; Tokio exposes tokio::signal::windows::ctrl_break() separately. Consequently EXIT_CODE_SIGTERM is unused on Windows, and the live windows-build-and-test check fails under -D warnings at line 1699. Handle both Windows event streams (and their registration errors) rather than suppressing the warning. The current let _ = tokio::signal::ctrl_c().await also treats registration failure as a received Ctrl-C and exits 130 immediately.

Blocking — required user-observable scenario is absent

tests/e2e-cucumber/features/dash.feature has no scenario covering this change. AGENTS.md §3 requires a Gherkin scenario for user-observable CLI behavior; the two unit tests only check constants and that a no-TTY helper call does not panic. The existing PTY harness already owns portable_pty::Child, whose process_id() API makes signal injection feasible. Add Linux PTY scenarios that deliver SIGTERM/SIGINT, assert 143/130, and verify the final output leaves the alternate screen and shows the cursor.

Spec

Blocking — a startup SIGTERM still reproduces the broken-terminal bug

crates/rocm-dash-tui/src/app/mod.rs:1610-1628

The code enters raw/alternate-screen mode before spawning an async task whose first poll registers the signal listeners. A signal received after EnterAlternateScreen but before that task is polled still takes the default disposition and skips restoration. In a PTY smoke test that sent SIGTERM as soon as \x1b[?1049h was observed, all 10/10 runs exited from the signal (-15) without LeaveAlternateScreen; delayed SIGTERM/SIGINT correctly exited 143/130 and emitted restoration sequences. Register the signal listeners synchronously before mutating terminal state, then move the initialized listeners into the watcher task. Propagate setup failure before entering raw mode; this also avoids permanently swallowing a successfully registered Unix signal if the second registration fails.

Should fix — implementation does not reuse the shared teardown helper

crates/rocm-dash-tui/src/app/mod.rs:1636-1646

The PR body says restore_terminal is reused by both signal and normal teardown paths, but normal return still duplicates the old teardown sequence. Either call the helper on the normal path or correct the PR description. One shared path avoids the two sequences drifting.

Verification

  • cargo test -p rocm-dash-tui: 686 passed; 4 ignored.
  • cargo clippy -p rocm-dash-tui --all-targets -- -D warnings: passed on Linux.
  • cargo build -p rocm: passed on Linux.
  • PTY delayed-signal smoke: SIGTERM → 143 and SIGINT → 130; both emitted LeaveAlternateScreen and Show.
  • PTY startup-race smoke: 10/10 immediate SIGTERM runs exited -15 without LeaveAlternateScreen.
  • Live CI: windows-build-and-test failed because EXIT_CODE_SIGTERM is unused; all other required completed checks passed at the reviewed head.

Decision: request changes.

r0x0r added a commit that referenced this pull request Aug 31, 2026
… (EAI-7194)

Address review on #326:

- Register the SIGTERM/SIGINT listeners BEFORE entering raw/alternate-screen
  mode. A signal delivered during startup is now latched by the already-installed
  handlers instead of taking the default disposition and killing the process
  mid-setup, which left the terminal in the broken raw/alt-screen state.
  Registration failure is propagated before any terminal state is mutated.

- Windows: subscribe to BOTH Ctrl-C and Ctrl-Break (tokio exposes them as
  separate streams; ctrl_c() alone misses Ctrl-Break). Ctrl-Break maps to the
  SIGTERM code and Ctrl-C to the SIGINT code, so EXIT_CODE_SIGTERM is now used on
  Windows and the -D warnings build is green. Registration failure is propagated
  rather than treated as a received Ctrl-C.

- Reuse the shared restore_terminal() helper on the normal-return path so the
  signal and clean-exit teardowns cannot drift.

- Add Linux PTY regression scenarios (dash.feature 11-13): deliver SIGTERM/SIGINT
  to the live dashboard and a startup-window SIGTERM at alt-screen entry, assert
  the 143/130 exit codes, and verify the terminal left the alternate screen and
  showed the cursor. Harness gains signal delivery (via kill(1) on the child pid),
  exit-code capture, and alt-screen/cursor restoration assertions.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the thorough review — addressed all four points in fd15c13:

Blocking — Windows Ctrl-Break / red windows-build-and-test. Replaced the single tokio::signal::ctrl_c() with a TerminationSignals type that registers both tokio::signal::windows::ctrl_c() and ctrl_break(). Ctrl-Break maps to the SIGTERM code and Ctrl-C to the SIGINT code, so EXIT_CODE_SIGTERM is now used on Windows and the -D warnings build is green. A failed registration is propagated (via ?) instead of the old let _ = ctrl_c().await that treated registration failure as a received Ctrl-C and exited 130 immediately.

Blocking (Spec) — startup SIGTERM race. The listeners are now installed before enable_raw_mode()/EnterAlternateScreen. Registration returns a TerminationSignals value that is moved into the watcher task, so a signal delivered any time after the terminal switches modes — even before the task's first poll — is latched by the already-installed handlers rather than taking the default disposition. Setup failure is propagated before any terminal state is mutated.

Blocking — required user-observable scenario. Added three Linux PTY scenarios in dash.feature (11–13): a steady-state SIGTERM (→143) and SIGINT (→130) to a running dashboard, plus a startup-window SIGTERM fired the instant the alternate screen is entered. Each asserts the conventional exit code and that the terminal left the alternate screen and showed the cursor. The harness gained signal delivery via kill(1) on Child::process_id(), exit-code capture, and alt-screen/cursor restoration assertions read from the vt100 screen.

Should fix — shared teardown. The normal-return path now calls the same restore_terminal() helper the signal path uses, so the two teardown sequences can't drift.

Local: cargo test -p rocm-dash-tui --lib (656 passed), cargo clippy -p rocm-dash-tui --all-targets -- -D warnings, and cargo clippy -p e2e-cucumber --test e2e -- -D warnings all pass. The new PTY scenarios are @requires-os:linux (no GPU needed) and run on the Linux E2E lanes; I can't execute them on macOS locally, so I'm watching CI to confirm.

@r0x0r

r0x0r commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

@michaelroy-amd all four points are addressed in fd15c13:

  • Windows Ctrl-Break / red build — replaced the single ctrl_c() with a TerminationSignals type that registers both windows::ctrl_c() and ctrl_break() (Ctrl-Break → SIGTERM code, Ctrl-C → SIGINT code), so EXIT_CODE_SIGTERM is used and -D warnings is green; registration failure now propagates via ? instead of being treated as a received Ctrl-C.
  • Startup SIGTERM race — listeners are registered before enable_raw_mode()/EnterAlternateScreen and the initialized value is moved into the watcher task, so a signal delivered before the first poll is latched rather than taking the default disposition; setup failure propagates before any terminal mutation.
  • Required scenario — added three Linux PTY scenarios in dash.feature (steady-state SIGTERM→143, SIGINT→130, and a startup-window SIGTERM), each asserting the exit code plus leaving the alternate screen and showing the cursor.
  • Shared teardown — the normal-return path now calls the same restore_terminal() helper as the signal path.

windows-build-and-test and all E2E lanes are green on the current head. Ready for a re-review — thanks.

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix itself is right and the revision history shows it — registering before raw mode, propagating setup failure before any terminal state is mutated, the shared restore_terminal helper, 128 + signo codes, and signal_task.abort() on a clean return. @michaelroy-amd's startup-race point is properly addressed, and the Ctrl-Break split is a genuine improvement over ctrl_c() alone.

I want to push on one thing @r0x0r raised and marked non-blocking. The observation was exactly right — tokio never unregisters the libc handler, so it survives signal_task.abort() — but the conclusion ("the process is exiting anyway, so this is fine to leave") holds only for rocm dash. On bare rocm it doesn't: dash::run_launcher is a synchronous loop that calls run_focused / run_chat / run, each of which builds a runtime, runs a session, drops the runtime, and returns to the loop. The process keeps living, in the launcher's own raw-mode/alt-screen event::read() loop, with a handler installed that catches SIGTERM and does nothing.

I built a probe mirroring that structure rather than reasoning about it. Control (no registration) exits 143 on the first SIGTERM. The PR-shaped version survives three SIGTERMs and a SIGINT and exits 0. Details in the first comment.

That makes bare rocm — the default invocation — unkillable by the default signals after the user's first flow, while sitting in exactly the raw-mode state this PR set out to protect. So I don't think it's leaveable, though the fix is small.

Two smaller things: scenario 13 doesn't reach the window it names (the polling granularity is ~1000× the race it guards), and the new restore_terminal unit test drives real global console state in the cargo test Windows lane. Also one stale doc link.

On @r0x0r's kill_on_drop question — worth answering explicitly in the code, and I'd note that whichever way it goes, process::exit in a peer task also skips run_async's embedded-daemon socket cleanup, so the two have the same shape.

Comment thread crates/rocm-dash-tui/src/app/mod.rs Outdated
// failure is propagated here, before any terminal state is mutated, so we
// never enter raw mode without a working restore path (and never silently
// swallow a signal because a second registration failed).
let termination = TerminationSignals::register()?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This registration is process-global and permanent, but it's scoped as if it were per-session — and on the bare-rocm path the process outlives the session.

dash::run_launcher (apps/rocm/src/dash.rs:359-377) is a synchronous loop:

loop {
    let serving = launcher_serving_instances(&paths);
    match rocm_dash_tui::ui::launcher::run_launcher(&theme, serving)? {
        None => return Ok(()),
        Some(choice) => match launcher_route(choice) {
            LauncherRoute::Focused(focus) => run_focused(focus)?,
            LauncherRoute::Chat => run_chat(chat_mock, ChatInferenceParams::default())?,
            LauncherRoute::Dashboard => run(None, false, chat_mock)?,
        },
    }
}

Each of those three builds a runtime, block_ons run_async → app::run, and drops the runtime on return. Tokio documents that the libc handler it installs is never removed (tokio/src/signal/unix.rs: "the libc signal handler is never unregistered"), and that handler doesn't terminate — it sets a flag and writes a byte to tokio's global pipe. After the runtime is gone there's no reactor draining that pipe and no listener, so the signal is caught and discarded.

Control returns to ui::launcher::run_launcher, which enters raw mode and the alt-screen itself (ui/launcher.rs:285-289) and blocks in event::read(). So the process sits in raw mode, permanently deaf to SIGTERM and SIGINT.

I built a probe with the same shape rather than trusting the reading:

########## CONTROL: no registration ##########
exit=143

########## PR SHAPE: register, drop runtime, keep looping ##########
session returned; runtime dropped
PID 47073 now in the synchronous outer loop
BUG: survived 5s of signals
exit=0

Three kill -TERM plus a kill -INT, all absorbed. The only difference between the two runs is whether a signal() stream was ever constructed.

Windows has the same shape for a different reason: SetConsoleCtrlHandler is installed once behind a OnceLock and the handler returns 1 ("handled") while the static watch::Sender lives, so Ctrl-C stops terminating the launcher too.

The practical effect: rocm → Set up → back at the menu → kill <pid> does nothing, and Ctrl-C doesn't either. That's the default invocation, and the terminal is in raw mode the whole time — a worse version of the bug this PR fixes.

This isn't a reason to revert; reverting leaves the handler installed too, since the current code also calls ctrl_c()/signal() inside the session. The registration needs to move up to where its lifetime matches the process: install the listeners once in apps/rocm's entry path with something alive to service them for the whole launcher loop, or keep a process-global watcher that the launcher also honours. A fallback in the synchronous launcher that restores the terminal and exits would work too, and might be simpler.

Worth a scenario as well — the existing PTY harness could drive rocm (not rocm dash), pick a flow, return to the menu, and then deliver SIGTERM. Scenarios 11/12 only cover the single-session path, which is why this is invisible today.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed at f6fe0ec — re-verified against the current head just now, not just re-read.

The watcher moved up to where its lifetime matches the process. dash::run_launcher (apps/rocm/src/dash.rs:376-387) now builds one multi_thread(1) runtime and spawns a single process-lifetime watcher on it via rocm_dash_tui::app::spawn_termination_watcher(), before the first menu paint. That runtime is held for the whole hub loop, so its worker keeps draining Tokio's signal pipe across every session runtime built and dropped inside the loop — the menu and each session alike stay killable. Registration happens inside the runtime context and the returned JoinHandle is dropped deliberately (drop detaches, it does not abort).

app::run (direct rocm dash, crates/rocm-dash-tui/src/app/mod.rs:1619) keeps its own watcher and abort()s it on the clean-return path (:1631) — that process dies with the session, so nothing changed there.

On the scenario you asked for: @id:dash-launcher-sigterm-restores-terminal-across-a-session (dash-13) drives exactly the shape you described — bare rocm, escalate into the dashboard (d), quit back to the menu (q) so that session's runtime is dropped, then SIGTERM — and asserts 143 plus alt-screen exit and cursor restored.

Verified on Linux at this head: cargo test --workspace --all-targets (0 failures), cargo clippy --workspace --all-targets -- -D warnings clean, and cargo xtask e2e with dash-11/dash-12/dash-13 all passing.

Then the dashboard exits from the signal with code 130
And the terminal is restored to the normal screen

@id:dash-startup-sigterm-restores-terminal @requires-os:linux

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario 13 can't observe the window it describes, so it doesn't guard the ordering fix.

The race is between enable_raw_mode() and the point the listeners become effective — microseconds. The scenario reaches it via:

  1. wait_for_alternate_screen, which polls at POLL_INTERVAL = 20ms (tui_driver.rs:51) until the parser has already consumed \x1b[?1049h;
  2. deliver_signal_and_wait, which fork/execs kill(1) — another process spawn.

By the time the signal lands, the event loop has been running for many milliseconds. So scenario 13 is behaviorally identical to scenario 11, and moving TerminationSignals::register() back below enable_raw_mode() would leave all three green. AGENTS.md §3 wants the regression test to fail before the fix.

Notably @michaelroy-amd's own smoke test did catch it (10/10 immediate SIGTERMs exited -15 without LeaveAlternateScreen) — but that harness armed on the raw byte stream, not on a 20ms poll of the parsed screen.

Two options that would work: have the product emit a startup marker before enable_raw_mode and arm a pre-forked helper on it, or cover the ordering with a unit test asserting registration precedes raw-mode entry. Dropping scenario 13 and saying in the PR text that the ordering is covered by construction is also honest — better than a scenario that reads as a guard and isn't one.

(Scenarios 11 and 12 are genuine: on main a signalled child surfaces as code=1 through portable_pty, not 143.)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and dropped at f6fe0ec. A 20 ms parser poll plus a fork/exec of kill(1) is orders of magnitude wider than the register-before-raw-mode window, so the old scenario 13 read as a guard without being one — moving TerminationSignals::register() back below enable_raw_mode() would indeed have left it green.

I took your third option plus a replacement rather than the marker/pre-forked-helper route:

  • The startup ordering is now covered by construction and said so explicitly in the PR text: spawn_termination_watcher() is called at crates/rocm-dash-tui/src/app/mod.rs:1619, enable_raw_mode()? at :1621, and registration is surfaced via ? before any terminal state is mutated. No PTY scenario is claimed for that microsecond window.
  • The freed-up dash-13 slot now holds the across-session launcher round-trip from your other comment (@id:dash-launcher-sigterm-restores-terminal-across-a-session), which is observable: open bare rocm, escalate to the dashboard, quit back to the menu (dropping that session's runtime), then SIGTERM → 143 + terminal restored. That one is genuinely fail-before/pass-after — with the hub watcher removed the SIGTERM step does not restore and the scenario is red.

dash-11/dash-12 are unchanged. All three pass under cargo xtask e2e on Linux at this head.

Comment thread crates/rocm-dash-tui/src/app/mod.rs Outdated
}

#[test]
fn restore_terminal_is_infallible_without_a_tty() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test mutates process-global terminal state, which is safe under nextest and not under cargo test.

restore_terminal() calls the real disable_raw_mode() and unconditionally writes \x1b[?1049l, the mouse-disable sequence, and \x1b[?25h to the process's shared stdout. On Windows disable_raw_mode is a SetConsoleMode on the shared console input handle; on Unix it's a tcsetattr guarded by crossterm's TERMINAL_MODE_PRIOR_RAW_MODE static.

The Linux lane runs cargo nextest run (ci.yml:328), which is process-per-test and isolates this completely. The required Windows lane runs cargo test --workspace --all-targets (ci.yml:357) — threads in one process — so this races every other test in the binary. It's the kind of thing that shows up as an unreproducible Windows-only flake much later.

It also writes escape sequences into the developer's shell when run locally under a real terminal.

Taking an impl Write for the escape-sequence half would let the test drive an in-memory sink and assert the emitted bytes — which is a stronger assertion than "didn't panic" anyway. The disable_raw_mode() call is the part that has to stay global, and it's also the part the test isn't really checking.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed at f6fe0ec, and the split is the one you suggested.

restore_terminal() (crates/rocm-dash-tui/src/app/mod.rs:1649) still calls the global disable_raw_mode() — that half genuinely has to be process-global and is not what the test was checking. The escape-sequence half is now write_restore_sequences<W: io::Write>(out: &mut W) (:1661).

The unit test (:3683, write_restore_sequences_leaves_alt_screen_and_shows_cursor) drives an in-memory Vec<u8> sink and asserts the actual emitted bytes — \x1b[?1049l and \x1b[?25h — instead of "didn't panic". It no longer touches disable_raw_mode(), the shared stdout, or the developer's live shell, so it is safe under the thread-per-test cargo test --workspace --all-targets Windows lane as well as the nextest Linux lane.

One caveat worth flagging since it is not in the diff you saw: the test is #[cfg(unix)]. On Windows crossterm drives the console through the WinAPI backend rather than emitting ANSI, and execute! into a Vec fails with "Initial console modes not set" — there is no console to configure. The reasoning is in a comment above the test. Production restore_terminal() passes a real stdout handle, so the Windows path is exercised there.

cargo test --workspace --all-targets passes on Linux at this head; windows-build-and-test is green in CI.

Comment thread crates/rocm-dash-tui/src/app/mod.rs Outdated
let signal_task = tokio::spawn(async move {
let code = termination.recv().await;
restore_terminal();
std::process::exit(code);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things follow from exiting inside the spawned task rather than unwinding through it, both minor on their own but they share a fix with @r0x0r's kill_on_drop question.

Socket leak. run_async (apps/rocm/src/dash.rs:662-672) aborts the embedded daemon and remove_files the socket only after app::run(args).await returns. process::exit never unwinds, so every SIGTERM/SIGINT leaves the socket path on disk. It self-heals — rocm-dash-daemon/src/server.rs unlinks a stale path on next bind — so the impact is small, but a previously-clean shutdown becomes a leaking one.

Concurrent-write race. The runtime is multi-threaded (dash.rs:222 new_multi_thread), so this task runs on a different worker than event_loop. It emits LeaveAlternateScreen + Show while the event loop may be mid-terminal.draw() on the same stdout. An interleaving of restore → draw → exit paints a partial dashboard frame into the user's normal-screen scrollback. Narrow, but it's the same class of corruption this PR fixes, and unlike the clean-exit path there's no ordering guarantee here.

Both go away with the same restructuring: make the signal a select! arm around event_loop so the loop unwinds first and the single existing teardown runs, then return the exit code up through run/run_async and process::exit at the top level. That also puts kill_on_drop back in play for @r0x0r's question — whichever answer you want, it becomes expressible rather than foreclosed.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially addressed at f6fe0ec — and I want to be precise about which part, because one of your three points is still open rather than fixed.

Socket leak and kill_on_drop: accepted and now documented, not fixed. spawn_termination_watcher's docs (crates/rocm-dash-tui/src/app/mod.rs:1686-1694) state the intent explicitly: on a signal we process::exit(128 + signo) and deliberately do not unwind, because SIGTERM/SIGINT means "stop now" and racing an orderly teardown against an imminent default-disposition kill is the worse trade. The two consequences are called out as intended — an in-flight focused install/serve child is left to the OS rather than reaped (matching the pre-existing default disposition, and avoiding truncating a mid-write child on the way out), and run_async's embedded-daemon socket is not unlinked here but self-heals on the daemon's next bind.

The concurrent-write race is still open. I did not fix it and, on re-reading, the doc comment enumerates only the other two consequences — that omission is on me and I am correcting the PR text.

The reason I did not take the select! restructuring is that it no longer composes with the fix for your other comment. The hub watcher now has to be process-lifetime (apps/rocm/src/dash.rs:376-387) precisely because bare rocm outlives each session runtime, and the launcher menu is a synchronous blocking event::read() — there is no future to select! against. So a select! arm inside app::run would fix the direct-rocm dash path while leaving the hub path racy, and worse, it would arm two independent handlers for the same signal during a session (the hub watcher plus the session's own arm), which is a new race rather than a fix.

The version that actually closes it is a process-global terminal-write lock shared by restore_terminal() and every render site — app::run's event loop, the focused host, the managers, and ui::launcher's synchronous loop. That is a cross-cutting change across the crate with its own deadlock-on-exit questions, and it is not something I want to land unreviewed on a green PR under the guise of addressing a review comment.

So: which do you prefer? (a) global write lock as its own follow-up, (b) some narrower serialization you have in mind, or (c) accept the window and document it honestly alongside the other two. I have taken (c) as the interim state and updated the PR body to say the window is known and open, so the PR text does not read as though this thread is settled. Happy to do (a) here or in a follow-up on your call — I would rather have your steer than guess at the layer.

Verified on Linux at this head: cargo test --workspace --all-targets (0 failures), cargo clippy --workspace --all-targets -- -D warnings clean, cargo xtask e2e with dash-11/dash-12/dash-13 passing.

/// observed exit code for the scenario's `Then` steps. Only harness faults
/// (no pid, a failed `kill`, a reader panic, or a timeout) are surfaced as
/// `Err`; asserting the exit *value* and terminal restoration is left to
/// [`expect_signal_exit_code`](Self::expect_signal_exit_code) and

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale intra-doc link: expect_signal_exit_code doesn't exist on TuiSession. The accessor is observed_exit_code (:609) and the assertion lives in the step at dash_steps.rs. (expect_terminal_restored on the next line is fine — that one's real, at :617.)

rustdoc isn't a CI gate here, so nothing catches it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed at f6fe0ec. deliver_signal_and_wait's rustdoc (tests/e2e-cucumber/tests/e2e/tui_driver.rs:537-542) now links [observed_exit_code](Self::observed_exit_code) (the real accessor, :575) and [expect_terminal_restored](Self::expect_terminal_restored) (:583), and says the exit-value assertion lives in the scenario's Then step. The nonexistent expect_signal_exit_code is gone — grep -n expect_signal_exit_code over the tree returns nothing at this head.

You are right that rustdoc is not a CI gate here, so nothing would have caught it.

@r0x0r
r0x0r force-pushed the fix/dash-restore-terminal-on-sigterm branch from fd15c13 to 2496202 Compare September 3, 2026 11:45
@r0x0r

r0x0r commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks @rominf, @michaelroy-amd, @r0x0r — rebased onto main and addressed every point. Force-pushed as a single squashed commit (2496202).

Blocking — bare rocm was unkillable across sessions (@rominf)

Your probe was exactly right: the per-session watcher registered inside app::run, but on the bare-rocm path the process outlives each session's runtime, and once that runtime drops nothing drains Tokio's signal pipe — so the synchronous launcher menu sat in raw mode, deaf to SIGTERM/SIGINT.

Moved the watcher up to where its lifetime matches the process. dash::run_launcher now builds one multi_thread(1) runtime and spawns a single process-lifetime watcher on it (rocm_dash_tui::app::spawn_termination_watcher) before the first menu paint, keeping that runtime alive for the whole hub loop. Its worker keeps draining the signal pipe across every session runtime built and dropped inside the loop, so both the menu and each session stay killable. The registration happens inside the runtime context; the returned JoinHandle is dropped on purpose (drop detaches, it doesn't abort), so the watcher lives for the process.

app::run (direct rocm dash) keeps its own watcher-with-abort() on the clean-return path — that path's process dies with the session, so nothing changed there.

Blocking — required scenario now exercises the across-session path (@rominf)

Replaced the ineffective startup-race scenario 13 with a launcher round-trip: open bare rocm, escalate into the dashboard (d), quit back to the menu (q) — dropping that session's runtime — then deliver SIGTERM and assert 143 + terminal restoration. This is the exact "back at the menu after a flow" state you described.

Verified fail-before/pass-after locally on this Linux host: with the hub watcher removed the scenario is red (the SIGTERM step hangs/does not restore), and green with it. Scenarios 11/12 (direct-dash SIGTERM/SIGINT) are unchanged and still pass.

scenario 13 couldn't observe its window (@rominf)

Agreed — a 20 ms parser poll is ~1000× the register-before-raw-mode race, so it read as a guard without being one. Dropped it. The ordering is now covered by construction: spawn_termination_watcher() is called before enable_raw_mode() in app::run, and registration is surfaced via ? before any terminal state is mutated. I didn't claim a PTY scenario for a microsecond window it can't see.

restore_terminal test mutated global console state (@rominf)

Split the helper: restore_terminal() still calls the global disable_raw_mode(), but the escape-sequence half is now write_restore_sequences<W: io::Write>(out). The unit test drives an in-memory Vec<u8> sink and asserts the emitted bytes (\x1b[?1049l, \x1b[?25h) — a stronger assertion than "didn't panic", and it no longer touches the shared terminal in the single-process cargo test Windows lane.

process::exit — socket leak, concurrent-write, kill_on_drop (@rominf, @r0x0r)

Stated the intent explicitly in spawn_termination_watcher's docs. On a signal we process::exit(128+signo) and deliberately do not unwind: SIGTERM/SIGINT means "stop now", so we restore the terminal and leave rather than race an orderly teardown against an imminent default-disposition kill. The two consequences are accepted as intended: (1) an in-flight focused install/serve child is left to the OS rather than reaped via kill_on_drop — matching the pre-existing default disposition and avoiding truncating a mid-write child on the way out; and (2) run_async's embedded-daemon socket isn't unlinked here, but the daemon unlinks a stale socket on its next bind, so it self-heals. As you noted, both share the same "peer task exits" shape.

stale doc link (@rominf)

Fixed deliver_signal_and_wait's rustdoc — it now points at observed_exit_code (the real accessor) and notes the assertion lives in the Then step, dropping the nonexistent expect_signal_exit_code.


Gates green locally: cargo clippy -p rocm-dash-tui -p rocm -p e2e-cucumber --all-targets -- -D warnings, cargo test -p rocm-dash-tui (656 passed), the three signal scenarios, and python3 scripts/smoke_local.py. The full pre-push gate (workspace clippy + test) passed on the force-push. Watching CI (incl. windows-build-and-test) to green.

@r0x0r
r0x0r requested a review from rominf September 3, 2026 11:46
@r0x0r
r0x0r force-pushed the fix/dash-restore-terminal-on-sigterm branch from 2496202 to d3ed2cd Compare September 3, 2026 13:33
… (EAI-7194)

A signal delivered to `rocm dash` (or bare `rocm`) tore the process down
while it still held the terminal in raw mode on the alternate screen, leaving
the shell needing `reset`. `app::run` now installs a termination-signal watcher
before entering raw mode that restores the terminal and exits `128 + signo`
(SIGINT -> 130, SIGTERM -> 143) on Unix (Ctrl-C/Ctrl-Break on Windows).

Bare `rocm` is a persistent hub: it outlives each session's Tokio runtime and
returns to a synchronous, raw-mode launcher menu between flows. Tokio never
unregisters the libc handler it installs, so once a session's runtime is
dropped nothing drains the signal pipe and a per-session watcher goes deaf —
leaving the menu holding the terminal in raw mode while SIGTERM/SIGINT are
caught-and-discarded (an unkillable, worse form of the bug). `run_launcher`
now keeps ONE runtime alive for the whole hub and spawns a single
process-lifetime watcher on it, so every window — menu and session alike —
stays killable.

The teardown is factored into `restore_terminal`, split so the escape-sequence
half (`write_restore_sequences`) takes an `impl Write` and a unit test asserts
the emitted bytes against an in-memory sink instead of mutating the process's
shared terminal state (which raced other tests in the single-process
`cargo test` lane). `spawn_termination_watcher` documents the deliberate
`process::exit` semantics: on a signal we stop promptly rather than racing an
orderly teardown, so an in-flight focused-install child is left to the OS
(matching the default disposition) and the embedded daemon's socket self-heals
on the next bind.

Tests: a Linux PTY scenario drives a full launcher round-trip (open the
dashboard, quit back to the menu, then SIGTERM) and asserts 143 plus terminal
restoration — red without the hub watcher, green with it. The dashboard
SIGTERM/SIGINT scenarios (11/12) are unchanged. The ineffective startup-race
scenario is dropped: a 20 ms-polled PTY cannot observe the microsecond
register-before-raw-mode window, which is instead covered by construction
(`spawn_termination_watcher` runs before `enable_raw_mode`).

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r
r0x0r force-pushed the fix/dash-restore-terminal-on-sigterm branch from d3ed2cd to f6fe0ec Compare September 3, 2026 13:55

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · f6fe0ec

Summary

Adds SIGTERM/SIGINT handling to the rocm dash TUI and the persistent bare-rocm launcher hub — a watcher task restores the terminal and exits 128 + signo — plus two unit tests and three PTY cucumber scenarios. Verdict: Needs work (the design is sound and well-reasoned; two concrete gaps). Verified: I ran the revert question on every added test — both new unit tests are non-discriminating (signal_exit_codes_follow_shell_convention restates the constants it asserts; write_restore_sequences_leaves_alt_screen_and_shows_cursor calls a leaf helper and would pass even if restore_terminal() were never wired to the watcher), but the three e2e scenarios genuinely discriminate — I confirmed from portable-pty 0.9.0 source (src/lib.rs:208-238) that a default-disposition signal kill yields exit_code() == 1, not 143, and that the vt100 parser still reports alternate_screen()==true on an unrestored kill, so dash-11/12/13 fail on unfixed code; I also confirmed @requires-os:linux is a real skip tag, dash-11/12/13 satisfy tests/e2e-cucumber/tests/feature_naming.rs, tokio features are correct, the #[cfg(windows)] block matches the real tokio API, rustfmt --check is clean on the changed file, and cargo check -p rocm compiles clean (my one permitted local run). On the red check: CI has a coverage gate (≥70% lines on the rocm-dash crates) and this PR adds ~90 lines of production logic to crates/rocm-dash-tui/src/app/mod.rs whose only real exercise is an out-of-process PTY subprocess that cargo llvm-cov cannot instrument — so a red check here is expected, not surprising, and points at the same test-discrimination gap below. Blocking: 2 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

  • crates/rocm-dash-tui/src/app/mod.rs:3667 — signal_exit_codes_follow_shell_convention asserts EXIT_CODE_SIGINT == 128 + 2 and EXIT_CODE_SIGTERM == 128 + 15 against two const i32 literals defined ~15 lines above (130, 143). It is a tautology: it would pass unchanged if spawn_termination_watcher() were never called from run(), if TerminationSignals::register() always failed, or if the spawned task never called restore_terminal(). Combined with write_restore_sequences_leaves_alt_screen_and_shows_cursor:3683 — which invokes the extracted helper directly and never touches restore_terminal(), spawn_termination_watcher, or run() — the entire in-process test surface for this feature protects nothing about the fix, and that is almost certainly what the coverage gate is red on. Concrete fix: delete the tautology and replace both with a test that exercises the real wiring in-process, e.g. #[tokio::test] async fn watcher_registers_and_aborts_cleanly() asserting spawn_termination_watcher() returns Ok, that the returned JoinHandle is still pending, and that abort() cancels it; and a test that drives TerminationSignals::recv() to completion for each kind and asserts it yields EXIT_CODE_SIGTERM / EXIT_CODE_SIGINT (deliverable in-process on Linux via a self-kill, which also makes the constants meaningful). That covers the registration/receive path the PTY subprocess cannot report coverage for.
  • crates/rocm-dash-tui/src/app/mod.rs:1619 + apps/rocm/src/dash.rs:385 — two watchers fire concurrently on the same signal. When bare rocm escalates from the hub into a session, app::run() registers a second TerminationSignals on the session runtime while the hub's process-lifetime watcher on signal_rt is still live. Tokio's signal registry is process-global (tokio/src/signal/registry.rs:89-105: one watch::Sender per signal, broadcast() notifies every subscriber regardless of which runtime registered it), so a real kill -TERM during a dashboard session — the most common path this PR targets — wakes both tasks: two threads race unsynchronized write_restore_sequences(&mut io::stdout()) calls and then both call std::process::exit(143). Concurrent exit(3) is undefined behaviour (glibc serialises on __exit_funcs_lock, so the loser blocks rather than returning), and the interleaved stdout writes can garble the restore sequences this PR exists to emit. The 30 lines of doc comment at dash.rs:363-375 and mod.rs:1683-1694 reason carefully about the hub-vs-session watcher lifetime but never note that the two coexist. Concrete fix: guard the exit path with a process-global static SHUTTING_DOWN: AtomicBool and have the watcher body return if swap(true, Ordering::SeqCst) was already true, so only the first watcher restores and exits; alternatively, have the hub pass a flag through ResolvedArgs so app::run() skips spawn_termination_watcher() when invoked from an already-watched hub.

Non-blocking

  • tests/e2e-cucumber/features/dash.feature:151 — # rominf regression: looks like a corrupted reference (sibling scenarios use a different form); worth fixing the word before it lands.
  • tests/e2e-cucumber/features/dash.feature:161-166 — dash-13 chains four When/And steps with no intervening Then, unlike every other scenario in the file; and its outcome steps say "the dashboard exits…" when the process under test is the launcher hub — rename the step or add a launcher-specific alias.
  • tests/e2e-cucumber/tests/e2e/tui_driver.rs:651-664 — drain_final_frame duplicates the drain loop already inlined at tui_driver.rs:422-440; extract one helper so the two cannot drift.
  • apps/rocm/src/main.rs:6076 (pre-existing, adjacent) — stream_attached_logs calls crossterm::terminal::enable_raw_mode() outside any Tokio runtime with no watcher, so its RawModeGuard::drop never runs on an external kill; same bug class this PR fixes, worth a follow-up.
  • apps/rocm/src/dash.rs:376 — the hub now unconditionally builds a second Tokio runtime (one extra worker thread) for every interactive bare-rocm invocation and keeps it alive alongside each session's own multi-thread runtime; documented and deliberate, just noting the cost.

Two watchers can be live at once: bare `rocm` escalates from the launcher
hub into a dashboard session, so the hub's process-lifetime watcher and
`app::run`'s session watcher coexist. Tokio's signal registry is
process-global, so a single `kill` wakes both — two threads then raced
unsynchronised `write_restore_sequences` writes to stdout and both called
`std::process::exit`.

Add a process-global `SHUTTING_DOWN: AtomicBool` claimed inside the
watcher body: only the first watcher through restores the terminal and
exits, the rest stand down. The latch is process-global rather than a
flag threaded through `ResolvedArgs` so it is path-independent — every
call site inherits it without knowing whether an outer watcher exists.

Replace the two non-discriminating unit tests with tests that exercise
the real wiring in-process:

- `termination_watcher_parks_until_aborted` — `spawn_termination_watcher`
  registers, the task stays parked while no signal has arrived, and
  `abort()` cancels it.
- `termination_signals_yield_shell_exit_codes_and_only_one_watcher_shuts_down`
  (Unix) — registers two `TerminationSignals`, raises one real signal,
  and asserts the first maps it to 143/130 while the second stands down.
  Safe to self-signal because it drives `await_termination` directly, so
  the body that calls `process::exit` never runs; serialised against the
  test above because the Windows lane runs `cargo test` (threads in one
  process), not `cargo nextest`.
- `only_the_first_caller_claims_the_shutdown_latch` — including under
  16-thread contention.

`await_termination` and `claim_shutdown` are split out of the watcher
body purely so the above can be asserted without `process::exit`.

Also, from review: restructure dash-13 into When/Then pairs and give the
launcher hub its own outcome step (the process under test is not a
dashboard), repair the corrupted scenario comment, and fold the
duplicated PTY drain loop into one `drain_final_frame` helper.

EAI-7194

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — both blocking points were real and are addressed in f971ab63.

One correction first. The review says the coverage gate was red and that this "points at the same test-discrimination gap". It wasn't: at f6fe0ec8, Coverage (rocm-dash crates, ratcheted) was pass (1m36s, job 100676754627), and the only non-green check in gh pr checks 326 was E2E tests (Strix Halo, Ubuntu), which the API reports as "conclusion": "cancelled" — a cancelled self-hosted hardware lane, not a failure, and not something the added unit tests could influence. So the supporting evidence doesn't hold. The substance of blocking #1 does hold on its own, and I've fixed it.

Blocking #1 — non-discriminating tests. signal_exit_codes_follow_shell_convention is deleted. In its place, three tests that drive the real wiring in-process:

  • termination_watcher_parks_until_aborted — spawn_termination_watcher() returns Ok, the returned JoinHandle is still pending after the task has been polled, and abort() cancels it (the clean-return path app::run uses).
  • termination_signals_yield_shell_exit_codes_and_only_one_watcher_shuts_down (#[cfg(unix)]) — registers two TerminationSignals, raises one real SIGTERM (then one SIGINT), and asserts the first receives it and maps it to 143/130 while the second stands down.
  • only_the_first_caller_claims_the_shutdown_latch — the latch below, including under 16-thread contention.

On the self-kill hazard you flagged indirectly: it is safe only because the test drives await_termination directly and never runs the watcher body that calls process::exit, and because the handlers are installed before the raise. Because the required Windows lane runs cargo test --workspace --all-targets (threads in one process) rather than cargo nextest, the signal tests are serialised by a module-local lock held across block_on — otherwise the live watcher in the first test would wake on the second test's signal and take the whole binary down with exit 143. Verified: 25/25 consecutive single-process cargo test -p rocm-dash-tui --lib runs, all green.

I kept write_restore_sequences_leaves_alt_screen_and_shows_cursor rather than deleting it. It is a leaf test, but it does discriminate (falsified below), and it is the only in-process cover for the restore payload itself; the wiring complaint is now answered by the three tests above.

Falsification — each new test was proven to fail against broken production code, then restored:

break red test
claim_shutdown returns true unconditionally only_the_first_caller_claims_the_shutdown_latch + …only_one_watcher_shuts_down
recv() maps SIGTERM→130, SIGINT→143 …only_one_watcher_shuts_down
watcher body returns instead of parking termination_watcher_parks_until_aborted
LeaveAlternateScreen dropped from write_restore_sequences write_restore_sequences_leaves_alt_screen_and_shows_cursor

Blocking #2 — two watchers on one signal. Confirmed, and fixed with the AtomicBool you suggested: a process-global SHUTTING_DOWN claimed inside spawn_termination_watcher's task body, so both call sites inherit it and only the first watcher restores and exits. Chosen over the ResolvedArgs flag because it is path-independent. The swap is extracted into claim_shutdown(&AtomicBool) — parameterised over the latch so the test uses a fresh one and stays order-independent — and the receive-then-arbitrate step into await_termination, so the whole path is testable without process::exit. The coexistence is now documented at both call sites.

Non-blocking — did these three:

  1. dash.feature:151 — corrupted comment repaired to the EAI-#### regression: form its siblings use.
  2. dash-13 restructured into When/Then pairs, and given a launcher-specific outcome step (the launcher exits from the signal with code …) plus a the launcher front door is displayed step, since the process under test is the hub, not a dashboard.
  3. tui_driver.rs — the inlined drain loop and drain_final_frame folded into one helper, drain_final_frame(Option<&str>). The no-marker path also gains the reader-panic check it was missing.

Deliberately not done:

  • Process-global terminal write lock for the watcher-vs-renderer race. This PR is scoped to watcher-vs-watcher; the wider window (watcher restoring while the renderer is mid-draw) needs a lock shared by restore_terminal and every render site, which is a larger change with its own deadlock surface. It stays a documented known limitation, and the PR description now says precisely which half is closed and which is not.
  • main.rs:6076 stream_attached_logs — agreed it is the same bug class, but it is pre-existing and adjacent; a follow-up, not this PR.
  • dash.rs:376 extra runtime — acknowledged as deliberate; the cost note stands, no change.

Verification (Linux, 112 cores, no GPU): cargo test --workspace --all-targets green; cargo clippy --workspace --all-targets -- -D warnings clean; cargo fmt --check clean; cargo xtask e2e — dash-01…dash-13 all pass (dash-13 now 9 steps). The one unexpected e2e failure on that box is serve-hf-checkpoint-inference, a GPU-serving scenario unrelated to this PR (that host has no GPU and it is green on the CI GPU lanes).

@r0x0r
r0x0r requested a review from siloteemu September 10, 2026 08:51
@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · f971ab6

Summary

Restores the terminal (leave alt-screen, show cursor, disable raw mode) and exits 128 + signo on SIGTERM/SIGINT for both rocm dash and the persistent bare-rocm launcher hub, with a process-global latch arbitrating the two watchers that now coexist. Approve. Verified: finding 1 FIXED — termination_signals_yield_shell_exit_codes_and_only_one_watcher_shuts_down (crates/rocm-dash-tui/src/app/mod.rs:3813) registers two real listener sets, raises a real signal, and asserts the first watcher maps it to 143/130 while the second returns None; it fails if await_termination drops the claim_shutdown arbitration or the exit-code mapping, and termination_watcher_parks_until_aborted (:3778) fails if spawn_termination_watcher stops registering or stops parking — plus three e2e scenarios (tests/e2e-cucumber/features/dash.feature:127-176) that assert the exact exit code and vt100 alt-screen/cursor state in the PR's default CI lane; finding 2 FIXED — the winner is decided by a single swap(true, SeqCst) and there is no .await between claiming the latch and restore_terminal(); process::exit(code), so the claim can never be abandoned by an abort or a runtime drop, and exactly one watcher writes and exits; the kept leaf test write_restore_sequences_leaves_alt_screen_and_shows_cursor (:3865) does discriminate (it fails if LeaveAlternateScreen or cursor::Show is dropped from the payload) and is worth keeping as the only in-process cover for the restore bytes, provided it is not counted as wiring cover — which the code comment correctly does not claim; in-process signal raising is safe here — TerminationSignals::register runs before every raise, Tokio never unregisters, so the fatal default disposition is unreachable, SIGNAL_TEST_LOCK is held across the whole block_on (not just up to an .await), the parking watcher is awaited to confirmed cancellation before the lock is released, and a repo-wide grep found no other test in this crate that registers signal handlers or spawns a watcher; on the revert question, every added test except the leaf payload test fails if the production change is reverted. Ran the three new unit tests locally with --test-threads=8: all pass. Blocking: 0 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

None.

Note on the prior round's evidence

The earlier review cited a red coverage gate as supporting evidence for finding 1. That was wrong, and you were right to correct it: at f6fe0ec8 the coverage gate concluded success, and the only non-green check was a self-hosted hardware lane whose conclusion was cancelled rather than failure. The cause was on our side — the tool used to collect check state reports a cancelled job as a failure, and that miscount was passed into the review as fact. It has been fixed to read conclusions directly. The substance of finding 1 stood on its own and you have addressed it; the corroboration should not have been there.

Non-blocking

  • crates/rocm-dash-tui/src/app/mod.rs:1705 — once any signal test runs, this test binary catches-and-discards SIGTERM/SIGINT for the rest of its life (nothing drains the pipe after the runtimes drop), so a hung cargo test -p rocm-dash-tui can no longer be Ctrl-C'd or SIGTERM'd, only SIGKILL'd. This is the same shape as the launcher bug the PR documents. Moving the signal tests into their own tests/ integration binary would contain it; introduced by the watcher registration itself, not by raise.
  • crates/rocm-dash-tui/src/app/mod.rs:3730 — SIGNAL_TEST_LOCK is enforced by a doc comment, not by construction. A future test in this crate that spawns a watcher without taking it would be taken down by the raise at :3835. No such test exists today; a shared fn with_signal_lock(...) wrapper would make it structural.
  • crates/rocm-dash-tui/src/app/mod.rs:1808 — the Windows Ctrl-C → 130 / Ctrl-Break → 143 mapping has no test cover: the arbitration test is #[cfg(unix)] and the e2e scenarios are @requires-os:linux. termination_watcher_parks_until_aborted covers registration and parking there, but not the codes. Adding Ctrl-Break (not just ctrl_c()) is a real behaviour change shipping untested.
  • tests/e2e-cucumber/tests/e2e/tui_driver.rs:627 — after the child is reaped, only the shared 250 ms DRAIN_TIMEOUT remains to consume the buffered restore sequences before the terminal-state assertion. The bytes are written synchronously before exit, so this should hold, but it is the single most plausible flake point in the three new scenarios; the constant is pre-existing and shared, so risk is not raised above the existing scenarios.
  • crates/rocm-dash-tui/src/app/mod.rs:1685 — the latch resolves watcher-versus-watcher, but a winning watcher's restore_terminal() still writes to stdout concurrently with the main thread's in-flight render. Inherent to restoring at signal time and outside the scope of the reported defect; noted so the remaining window is explicit rather than assumed closed.

rominf
rominf previously requested changes Sep 10, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This head resolves nearly everything, and I checked the resolutions against the code rather than the replies: Michael's four points, four of my five, and both of Teemu's. I traced await_termination specifically and confirmed the SHUTTING_DOWN latch leaves no cancellation window between claiming it and process::exit, so exactly one watcher restores and exits. Replacing the tautological test with three that drive the real wiring was the right move.

Two things below. One is the question you put to me, which I owe you an answer on; the other is a coverage claim I think is wrong on both platforms rather than just Windows.

Also worth recording: async-signal-safety isn't an issue here, since the only libc handler in play is tokio's write-to-a-self-pipe and the restore work runs in ordinary task context. I looked for that and didn't find a problem.

One non-blocking note I'm not asking you to act on: the new signal tests leave tokio's handlers installed process-wide, and tokio never unregisters them, so for the remainder of a single-process cargo test run the binary is deaf to SIGINT/SIGTERM. That's the same class of global-test-state I objected to earlier arriving by a different route, and it's relevant because the required Windows lane runs cargo test rather than nextest. A line in the SIGNAL_TEST_LOCK comment would be enough.

/// watcher even though the [`Terminal`] backend owns its own `stdout` clone),
/// then writes the alt-screen/mouse/cursor restore sequences to a fresh
/// `stdout`. Every step is best-effort: on a signal we are about to exit anyway,
/// and a vanished controlling terminal must not turn teardown into a panic.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Answering the question you left me on the watcher-versus-renderer window, since that's a reviewer call rather than something you should have to guess at.

I don't think a process-global terminal-write lock is worth it here. But I'm also not ready to just accept the window, because its consequence depends on something the comment doesn't state, and the two outcomes are very different.

If the renderer can only be interleaved during the restore — some escape sequences arriving out of order on a terminal that's about to be reset anyway — then it's cosmetic, and I'd accept it with a comment saying so.

If the renderer can complete a draw after the watcher has finished restoring, then it re-emits the alternate-screen sequences and undoes the restore. That leaves exactly the broken terminal this PR exists to fix, just through a narrower door and only sometimes — which is worse to diagnose than the original bug.

So: can a draw land after restore_terminal() returns? If it can't, say why in the comment and I'm happy. If it can, I'd rather see the narrow serialization than the global lock — the renderer checking the same SHUTTING_DOWN latch before it draws would close it without introducing a lock on every terminal write.

/// Park until a termination signal arrives, then arbitrate on `latch`.
///
/// Returns `Some(128 + signo)` for the single watcher that wins the latch — that
/// caller must restore the terminal and exit with the code — and `None` for any

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is right, and it makes the PR body's Windows claim broader than what the code delivers.

crossterm's raw mode clears ENABLE_PROCESSED_INPUT. With that flag off, the Win32 console delivers Ctrl+C into the input buffer as a key event and does not raise CTRL_C_EVENT — so tokio::signal::windows::ctrl_c() never fires. CTRL_BREAK_EVENT is unaffected, so the Ctrl-Break arm genuinely works. Since spawn_termination_watcher() is two lines above enable_raw_mode(), the ctrl_c() arm is only live for that startup window, which the dash.feature comment already acknowledges a polled PTY scenario can't observe.

The part I'd extend beyond Windows: the same reasoning applies on Linux, where raw mode clears ISIG and a typed Ctrl-C therefore doesn't generate SIGINT either. Three things point the same way — dash-12's comment calls SIGINT "the interactive Ctrl-C gesture" but deliver_signal_and_wait shells out to kill(1), so no scenario ever sends a literal 0x03; there's no KeyModifiers::CONTROL handling in app/mod.rs or ui/launcher.rs; and the one Ctrl+C handler that does exist in this crate, ui/job_console.rs:47, treats it as a key event, which is what raw mode makes it.

If that holds, SIGTERM and an externally delivered kill -INT work exactly as this PR claims — but typing Ctrl-C in the dashboard or the launcher does nothing at all on either platform, which is probably what a user would actually try first.

I couldn't run this, so please treat it as a question rather than a verdict: send 0x03 through the existing PTY driver and see whether the process restores and exits. If it doesn't, the fix is to handle KeyCode::Char('c') with CONTROL in both key loops and route it through the same restore path, keeping the ctrl_c() stream for the pre-raw-mode window. Either way the doc comment and the PR body want narrowing to Ctrl-Break.

…g the restore

Raw mode clears ISIG (Unix) and ENABLE_PROCESSED_INPUT (Windows), so while the
TUI is up a typed Ctrl-C is never turned into a SIGINT or a CTRL_C_EVENT: the
process receives 0x03 as an ordinary key event. Neither key loop handled it, so
the first gesture a user reaches for did nothing at all and left them holding a
raw-mode terminal. Measured through the PTY driver on Linux: the dashboard and
the launcher both ignored 0x03 indefinitely — no exit, still on the alternate
screen, cursor still hidden — and only a SIGTERM got them out. The same flag is
why tokio's windows ctrl_c() stream cannot fire while the TUI is up, leaving
Ctrl-Break as the only console arm that genuinely works there.

Handle it as a key in both of the process's key loops — the dashboard event loop
and the launcher's synchronous menu — routed through the same restore-and-exit
path a real SIGINT takes, so the gesture cannot mean one thing inside a session
and nothing at the front door. It sits above every overlay so no screen can trap
it; a displayed job console keeps its established Ctrl+C meaning of cancelling
the running job rather than killing the process mid-write.

Also close the watcher-versus-renderer window instead of documenting it as a
limitation. A draw can complete after restore_terminal() returns — the watcher
runs on a Tokio worker thread while frames are drawn on the thread owning the
loop, and nothing locks the terminal. It cannot re-enter the alternate screen,
but every frame ends by hiding the cursor and repainting, so a late one would
undo half the restore and paint a stale dashboard over the restored screen. In
preference to a process-global terminal-write lock, both render loops now check
the same SHUTTING_DOWN latch before drawing; the latch is claimed before the
restore begins, so no frame can start after the claim. A frame already in flight
can still interleave during the restore — cosmetic, and now stated as such.

Narrow the signal doc comments and the SIGINT scenario to what they actually
cover, and record on SIGNAL_TEST_LOCK that Tokio never unregisters the handler
these tests install, so a single-process cargo test run is deaf to those signals
from the first of them onward.

dash-14 and dash-15 send a literal 0x03 through the existing PTY driver and
assert terminal restore plus exit 130, for the dashboard and the launcher.

EAI-7194

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed 4639efbf. You were right on both counts, and the second one turned out to be the one that mattered.

The 0x03 experiment. Ran it on Linux through a controlling PTY (pty.fork + TIOCSWINSZ, the same shape as the harness's openpty(PtySize)), with two controls first: 0x03 to a cooked-mode child is delivered as SIGINT (WIFSIGNALED, sig 2), so the harness really does send the byte; and q to the running dashboard exits 0 with the full restore, so writes to the master really do reach the raw-mode app. Then the real thing, against f971ab63:

  • rocm dash --demo, marker Updates seen, alt-screen entered, send 0x03 → child did not exit within 15 s, 2500 bytes emitted after the send, ESC[?1049l never appeared. A subsequent SIGTERM got it out with 143.
  • bare rocm at the front door, send 0x03 → did not exit within 15 s, 25 bytes emitted, no ESC[?1049l. SIGTERM → 143.

So: hung, alt-screen never left, and the only thing that worked was the signal path this PR already fixes. Your reasoning holds exactly — including the three supporting points, all of which check out (deliver_signal_and_wait shells out to kill(1), there was no KeyModifiers::CONTROL anywhere in app/mod.rs or ui/launcher.rs, and the one Ctrl+C handler in the crate is job_console.rs treating it as a key). I did not run anything on Windows; the ENABLE_PROCESSED_INPUT half is reasoned from the documented semantics and now says so in the doc comment instead of overclaiming.

Fixed as you proposed: KeyCode::Char('c') + CONTROL in both key loops, routed through one exit_on_ctrl_c that claims the same latch, restores, and exits 130. Above every overlay so nothing traps it — with one exception I want to flag rather than bury: while a job console is displayed, job_console::on_console_key already gives Ctrl+C the meaning "cancel this running job", which focused_close_key_blocked documents as the way to stop a focused install/serve without truncating it mid-write. Killing the process there would have been a regression, so that meaning wins and ctrl_c_should_exit encodes the precedence. dash-14/dash-15 send the literal 0x03 and assert restore + 130 for the dashboard and the launcher; both were verified red with the two handlers removed (the wait times out) and green with them. Post-fix probe: both surfaces exit 130 with ESC[?1049l ... ESC[?25h.

Can a draw land after restore_terminal() returns? Yes — so I implemented the narrow serialization rather than accepting the window. The paths: the watcher is tokio::spawned on a multi-thread runtime (build_dashboard_runtime, and multi_thread(1) for the hub in dash::run_launcher), while frames are drawn on the thread that owns the loop — rt.block_on's thread for event_loop, and the main thread for launcher::run_launcher, which is synchronous around a blocking event::read(). Two real OS threads, nothing locking the terminal, and no await point in restore_terminal(); process::exit(code) to serialize them.

The consequence is narrower than "undoes the restore" but wider than cosmetic. EnterAlternateScreen is emitted exactly once, in run/run_launcher before the loop, and Terminal::draw never re-emits it — so a late frame cannot put you back on the alt-screen. But draw ends in hide_cursor (or show_cursor + set_cursor_position) and a full repaint, so a frame completing after the restore re-hides the cursor and paints a stale dashboard onto the normal screen: exactly the cursor_hidden half of what expect_terminal_restored checks. So both loops now check app::shutdown_claimed() before drawing. The latch is claimed before the restore starts, so no frame can begin after the claim; a frame already in flight when the claim lands can still interleave during the restore, and that one is genuinely cosmetic — reset in the same breath, restore bytes written last — and the comment now says that rather than leaving it implicit.

Also took the non-blocking note: SIGNAL_TEST_LOCK now records that Tokio never unregisters the handler register() installs, so from the first of these tests onward a single-process cargo test run is deaf to SIGINT/SIGTERM, and future tests must not assume the default disposition survives. The PR body is narrowed to Ctrl-Break for the Windows console path throughout.

Verification: cargo test --workspace --all-targets and cargo clippy --workspace --all-targets -- -D warnings clean; cargo xtask e2e on a GPU-less Linux box is green apart from serve-vllm-inference / serve-hf-checkpoint-inference, which need a GPU and pass on the self-hosted lanes. Each new unit test was falsified individually (dropping the modifier check from is_ctrl_c, dropping the console exception from ctrl_c_should_exit, and making the render gate ignore the latch turn exactly those three red and nothing else). Leaving the threads for you.

@r0x0r
r0x0r requested a review from rominf September 10, 2026 12:04
@siloteemu

siloteemu commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 4639efb

This automation posts comments only. It never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Restores the terminal on SIGTERM/SIGINT across both the dashboard session and the persistent bare-rocm launcher hub, and — at this head — also handles a typed Ctrl-C as a key event (raw mode suppresses the signal) and gates both render loops on a shutdown latch so a late frame cannot undo the restore. Verified: ran cargo test -p rocm-dash-tui --lib filtered to the new signal/ctrl-c/restore tests — 7 passed, 0 failed; separately confirmed by reading source that spawn_termination_watcher() precedes enable_raw_mode() in app::run, that both production terminal.draw sites are latch-gated, that both key loops route Ctrl-C through exit_on_ctrl_c, that the e2e Ctrl-C steps send a literal 0x03 byte rather than a signal, that expect_signal_exit_code no longer appears anywhere in the tree, and that libc resolves from the workspace as a unix-only dev-dependency. No prompt-injection content found in the diff or comments. Blocking: 0 · Non-blocking: 5.

Status of the existing change requests

I am judging code only; I did not contact GitHub, so I cannot tell whether either reviewer has formally dismissed or re-affirmed their review — only whether the code at this head answers what they wrote.

michaelroy-amd (review at 4048d69d)

  1. "Windows build is red and Ctrl-Break is not handled" — asked that both Windows event streams be registered with their registration errors handled, rather than suppressing the unused-constant warning, and objected that let _ = ctrl_c().await treats a registration failure as a received Ctrl-C. Appears addressed. The #[cfg(windows)] TerminationSignals now registers ctrl_c() and ctrl_break() as separate streams, both via ?, and recv() selects across them mapping Ctrl-Break → EXIT_CODE_SIGTERM and Ctrl-C → EXIT_CODE_SIGINT, so the constant is used on Windows too. I cannot compile for Windows here, so the build-goes-green half is inference from the code, not a measurement.
  2. "Required user-observable scenario is absent" (AGENTS.md §3; suggested using portable_pty::Child::process_id() for signal injection, asserting 143/130 and alt-screen exit + cursor shown) — appears addressed. dash.feature gains five scenarios (dash-11..dash-15); the driver's deliver_signal_and_wait takes the child's pid and shells out to kill(1), and expect_terminal_restored() asserts the vt100 parser shows neither alternate screen nor hidden cursor.
  3. "A startup SIGTERM still reproduces the broken-terminal bug" — asked that listeners be registered synchronously before terminal state is mutated and that setup failure propagate first. Appears addressed. spawn_termination_watcher()? is the first statement in app::run, two lines above enable_raw_mode(), and TerminationSignals::register propagates both registrations via ?. There is deliberately no PTY scenario for that microsecond window — the author says so in the dash-13 comment and in the PR text rather than claiming coverage; the other reviewer had agreed that is the honest option.
  4. "Should fix — implementation does not reuse the shared teardown helper" — appears addressed. The clean-return path in run now calls the same restore_terminal() the signal path calls, so the two sequences cannot drift.

rominf (first review, fd15c13c)

  1. Process-global registration scoped as if per-session; bare rocm outlives the session and goes deaf/unkillable in raw mode. Appears addressed, and by the route the reviewer named ("install the listeners once in the entry path with something alive to service them"): run_launcher builds one worker_threads(1) runtime held for the whole hub loop and spawns a single detached process-lifetime watcher on it before the first menu paint. The requested scenario exists as dash-13, which round-trips into a session and back before signalling.
  2. Scenario 13 cannot observe the window it names. Appears addressed by the third option the reviewer offered — the old scenario was dropped, the ordering is stated as covered by construction, and the freed slot now holds the across-session hub scenario.
  3. restore_terminal unit test drives real global console state, unsafe under the cargo test Windows lane. Appears addressed exactly as suggested: the escape-sequence half is now write_restore_sequences<W: io::Write>, the test drives a Vec<u8> sink and asserts the actual \x1b[?1049l / \x1b[?25h bytes, and the global disable_raw_mode() stays outside it.
  4. Socket leak, kill_on_drop, and the concurrent-write race from exiting inside the spawned task. Split outcome, both parts now settled. The socket leak and the unreaped child are accepted and documented as intended behaviour in spawn_termination_watcher's docs — that is a judgement call the reviewer may still disagree with, and it is recorded rather than fixed. The concurrent-write race was open at the previous head and is closed at this head (see the next section).
  5. Stale intra-doc link expect_signal_exit_code. Addressed — grep -rn expect_signal_exit_code over the tree returns nothing, and the rustdoc now links observed_exit_code and expect_terminal_restored.

rominf (second review, f971ab63)

  1. "Can a draw land after restore_terminal() returns?" — the reviewer asked for either an explanation in the comment (if it cannot) or a narrow serialization (if it can), explicitly preferring the renderer checking the same latch over a global write lock. Appears addressed, and by the preferred route. The answer given is "yes it can", and both production render loops now consult shutdown_claimed() before drawing, with the latch claimed before the restore begins. I verified there are exactly two production terminal.draw sites and both are gated. The residue the reviewer would have accepted as cosmetic — a frame already in flight when the claim lands — is stated as accepted in restore_terminal's doc comment.
  2. Typed Ctrl-C never becomes a signal under raw mode, on Windows and Linux; PR's Windows claim is too broad. Appears addressed. Ctrl-C is now handled as a key in both key loops and routed to the same restore-and-exit path; the Windows doc comment is narrowed to say the ctrl_c() arm cannot fire while the TUI is up and that Ctrl-Break is the arm that genuinely works; dash-14/dash-15 send the literal 0x03 byte through the PTY rather than a signal. The reviewer asked this be treated as a question to be measured — the commit message states it was measured through the PTY driver on Linux; I could not re-run the e2e suite here, so that measurement is the author's, not mine.
  3. Non-blocking note: record in the SIGNAL_TEST_LOCK comment that tokio's handlers stay installed process-wide for the rest of a cargo test run. Appears addressed — the lock's doc comment now has a paragraph saying exactly that, including the warning for any future test that expects a signal to kill the test binary.

Nothing above looks to me like a point that was quietly dropped. The one item that is answered by argument rather than by code is michaelroy-amd's startup-race coverage (no scenario for the microsecond window) and rominf's #4 socket-leak half — both are stated openly in the code and PR text rather than papered over, which is a reviewer call rather than a defect.

Anything new wrong at this head? The code pushed after those reviews is the single commit that handles a typed Ctrl-C as a key and gates both render loops on the shutdown latch. I found no defect in it that rises to blocking. Two things in it are worth the author's eye and are listed below: a comment the change itself made false, and one terminal-mutating site the new latch does not guard. Neither is a regression against the previous head.

Measured check state at this head: 25 success, 2 skipped, 0 failure, 0 pending. I have no per-lane detail and cannot attribute any of these to a named job.

🚫 Blocking (must fix before merge)

None.

Non-blocking

  • crates/rocm-dash-tui/src/app/mod.rs:2261 — the approval-modal arm's comment still says the modal "owns ALL keys with the highest priority (above every operational overlay and the general handler)", but the new Ctrl-C arm at :2258 now sits above it and ctrl_c_should_exit does not consult state.approval. The behaviour change looks deliberate and safe (Ctrl-C exits, it never approves), but the comment is now false and is exactly the kind of thing the next reader will trust; narrowing it to "owns all keys except a typed Ctrl-C" is a one-line fix.
  • crates/rocm-dash-tui/src/ui/launcher.rs:285-287 — each hub iteration re-enters raw mode and the alternate screen with no shutdown_claimed() check, unlike the draw twelve lines below. A signal landing in the gap between a session's clean return and this re-entry lets the hub watcher restore and process::exit while the main thread is re-entering. Narrow, and strictly better than the pre-PR behaviour rather than a regression — but it is the one terminal-mutating site the latch does not guard.
  • crates/rocm-dash-tui/src/app/mod.rs:3853+ — of the seven new unit tests, four (claiming_the_shutdown_latch_suspends_rendering, ctrl_c_is_recognised_as_a_key_..., ctrl_c_exits_the_session_but_a_job_console_..., write_restore_sequences_...) call the leaf helpers directly and would still pass with the corresponding call sites deleted from event_loop, run, and launcher.rs; the other three exercise real registration/arbitration but likewise never reach run(). The wiring regression coverage rests entirely on dash-13/dash-14/dash-15, which do fail-before/pass-after by timeout. That is a legitimate split given the exit path cannot be unit-tested, but it is worth knowing that a wiring regression is caught only by the e2e lane.
  • tests/e2e-cucumber/tests/e2e/dash_steps.rs:111 — the shared helper is named signal_dashboard but is also what #[when("the launcher receives a SIGTERM")] calls; the file is otherwise careful to keep launcher and dashboard wording distinct, so a neutral name would match that intent.
  • tests/e2e-cucumber/tests/e2e/dash_steps.rs:15 — the blank line between the use block and the following doc comment was removed, leaving the import butted against unrelated documentation.

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — approving. Nice work on the signal handling and shutdown-latch coverage.

A few things worth fixing, but none of them block this merge:

  1. Ctrl-C bypasses an open approval modal. In event_loop, the new ctrl_c_should_exit arm exempts an active job console but not an open approval modal, so Ctrl-C while the modal is up currently force-exits instead of routing to the modal handler. The modal's doc comment ("owns ALL keys above every operational overlay") is now stale as a result. Worth either adding a modal exemption (mirroring the job-console one) or, if the force-exit is intentional, updating the comment and adding a test for "Ctrl-C while approval modal open."

  2. Windows risk disclosure is slightly incomplete. The stated reasoning checks out, but it doesn't mention that TerminationSignals never registers CTRL_CLOSE_EVENT/CTRL_LOGOFF_EVENT/CTRL_SHUTDOWN_EVENT (pre-existing, not introduced here), and that register() currently only has Unix unit test coverage. Worth expanding the disclosure or filing a follow-up issue.

  3. Test naming nit: claiming_the_shutdown_latch_suspends_rendering exercises the atomic latch sequentially rather than racing a concurrent draw. The underlying race looks benign, but the name overclaims what's being tested — consider a rename or extending it to a real two-thread scenario.

Approving as-is; happy to see these addressed in a fast-follow.

@r0x0r
r0x0r dismissed stale reviews from michaelroy-amd and rominf September 15, 2026 08:18

addressed

@r0x0r
r0x0r added this pull request to the merge queue Sep 15, 2026
@r0x0r

r0x0r commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

Worked through the automated round (5 non-blocking findings against 7a82ef8d, which was two commits stale by the time it landed). Checked all five against the current head; pushed f3e29d6e for the three that still held.

Already fixed by the rominf round, no action taken:

  • Duplicate on_alternate_screen — already deleted; only in_alternate_screen remains (tui_driver.rs:672).
  • Render-gate coverage — the gate was extracted into draw_frame_unless_shutting_down (app/mod.rs) and draw_menu_unless_shutting_down (ui/launcher.rs), called from event_loop and run_launcher respectively, with TestBackend coverage on both sides. Considered addressed.

Fixed in f3e29d6e:

  1. Stale comment above the approval-modal arm. It claimed the modal "owns ALL keys with the highest priority". The Ctrl-C arm now sits above it, so that absolute is no longer true. Reworded to say the modal owns every remaining key and that only the Ctrl-C arm outranks it; the rationale for that ordering stays where it already lives, in the arm itself.

  2. Real coverage gap in write_restore_sequences_leaves_alt_screen_and_shows_cursor. It asserted \x1b[?1049l and \x1b[?25h, but the writer emits three commands — LeaveAlternateScreen, DisableMouseCapture, cursor::Show — so deleting DisableMouseCapture left the test green while leaving the terminal reporting mouse events after exit. Escape bytes were derived from a sink run rather than guessed; crossterm 0.28 emits:

    "\u{1b}[?1049l\u{1b}[?1006l\u{1b}[?1015l\u{1b}[?1003l\u{1b}[?1002l\u{1b}[?1000l\u{1b}[?25h"
    

    DisableMouseCapture is the middle five modes (SGR-encoding, urxvt-encoding, any-motion, button-event, normal tracking). The test now asserts that whole block, and is renamed ..._leaves_alt_screen_disables_mouse_and_shows_cursor.

    Falsification — dropped DisableMouseCapture from the writer and re-ran:

    test app::tests::write_restore_sequences_leaves_alt_screen_disables_mouse_and_shows_cursor ... FAILED
    
    panicked at crates/rocm-dash-tui/src/app/mod.rs:4468:9:
    expected the disable-mouse-capture sequences
    "\u{1b}[?1006l\u{1b}[?1015l\u{1b}[?1003l\u{1b}[?1002l\u{1b}[?1000l"
    in "\u{1b}[?1049l\u{1b}[?25h"
    

    Restored the command, test green again. The new literal is a let inside the already-#[cfg(unix)] test body, not a module-level const, so it cannot become dead code on Windows — verified by flipping the test to #[cfg(windows)] and running Linux clippy, which compiles the same shape the Windows lane does: clean.

  3. signal_task.abort() comment overstated its guarantee. It read as though abort stops the watcher firing after a clean return. spawn_termination_watcher's body goes straight from await_termination(...).await into std::process::exit(code) with no await point in between, so once the signal has resolved there is nowhere for the cancellation to land — a signal arriving in the instant before the abort still exits 128 + signo rather than returning res. Added the same "narrow window, accepted" acknowledgement the frame race gets elsewhere in the file; behaviour unchanged, and the terminal is restored on both paths either way.

Gates on the Linux box: cargo test -p rocm-dash-tui --all-targets green, cargo clippy --workspace --all-targets -- -D warnings clean, cargo fmt --all --check clean. Nothing under tests/e2e-cucumber was touched.

Two regressions from e3e36cc, both surfaced by the E2E lane.

`run`'s clean-quit teardown claimed `SHUTTING_DOWN`. That latch is a one-shot
*process-exit* arbiter — nothing releases it, because every other claimer goes
straight on to `process::exit`. `run` returning is the one teardown with an
"after": bare `rocm` is a persistent hub, so control goes back to a live
launcher menu. Claiming there wedged the hub three ways at once — the render
gate refused every subsequent frame (`dash-14` timed out on a blank front door),
`exit_on_ctrl_c` parked forever, and every later `await_termination` lost the
claim and returned without exiting, so the hub silently swallowed SIGTERM: the
unkillable-terminal failure this PR exists to fix, reintroduced one layer up.

So the exit latch is read, not claimed, by `restore_after_session`: if a watcher
or a typed Ctrl-C already owns the exit it is microseconds from ending the
process and a second restore is redundant; otherwise the session restores and
leaves the latch for the windows that follow. rominf's finding stands — the
single-writer guarantee moves to `RESTORE_LOCK`, a mutex held across
`restore_terminal`'s escape bytes, which is strictly stronger than the latch
was: it covers the two-watchers-on-two-runtimes case too, and it holds even
though the check above is racy on its own. The render gate keeps its meaning,
now an exact one — "the process is exiting".

Second, the help modal was sized to a fixed 70% of the body. At the 80x24 the
e2e lane pins that is 14 rows; the new Ctrl-C hint wraps to two lines at that
width, which pushed the per-tab guidance off the bottom with no scrollbar or
indicator — `dash-05` failed on "Home tab", text unrelated to the hint that was
added. The hint is documented deliberately and stays; the modal is now sized to
the height `Paragraph::line_count` says its wrapped content needs, clamped to
the area, so a new hint can no longer silently evict an unrelated one.
`centered_width` is split out of `centered_rect` so the pre-layout width cannot
drift from the real geometry.

Falsified: restoring `centered_rect(70, 70, 80, 22, area)` reds
`help_modal_shows_every_hint_at_the_minimum_supported_geometry`, printing the
same modal-ends-at-`{ / }` render the failing CI job captured; restoring
`claim_shutdown` in `restore_after_session` reds both
`a_clean_session_teardown_restores_without_claiming_the_exit_latch` and
`the_front_door_comes_back_after_a_session_ends_cleanly`, the latter on a blank
buffer — the e2e symptom, at unit cost. `RESTORE_LOCK`'s mutual exclusion has no
deterministic single-threaded observable and is not separately tested.

All 16 `dash` e2e scenarios pass on the mock lane.

Reported-by: rominf
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

Both E2E tests failures were real, both introduced by e3e36cc, both fixed in 94702d1.

1. dash-14 — the launcher hub wedged itself after the first session

e3e36cc made run's clean-quit teardown claim SHUTTING_DOWN, to close the double-restore-write race you flagged. But SHUTTING_DOWN is a one-shot process-exit arbiter: nothing releases it, because every other claimer goes straight on to process::exit. run returning is the one teardown with an "after" — bare rocm is a persistent hub, so control goes back to a live launcher menu.

Claiming there wedged three things at once, all invisible to the unit suite:

  1. draw_menu_unless_shutting_down saw the claimed latch and refused every subsequent frame — a blank front door. That is the reported failure: timed out after 30s waiting for "Set up this system".
  2. exit_on_ctrl_c parked forever, so Ctrl-C at the front door did nothing after any session.
  3. Every later await_termination lost the claim and returned without exiting — the hub silently swallowed SIGTERM. That is the unkillable-terminal bug this PR exists to fix, reintroduced one layer up.

Fix: the exit latch is now read, not claimed, by a named restore_after_session. If a watcher or a typed Ctrl-C already owns the exit, it is microseconds from ending the process and a second restore is pure redundancy; otherwise the session restores and leaves the latch alone for the windows that follow.

Your finding stands — the single-writer guarantee moves to RESTORE_LOCK, a mutex held across restore_terminal's escape bytes. That is strictly stronger than the latch was: it also covers two watchers on two runtimes, and it holds even though the read above is racy on its own (abort() cannot stop a watcher already resumed past its .await). It is held for a handful of bytes, so it adds no hang profile beyond the internal lock io::stdout() already takes on every write. The render gate keeps its meaning and gains an exact one: "the process is exiting".

2. dash-05 — the help modal silently truncated

The modal was sized to a fixed 70% of the body rect. At the 80x24 the e2e lane pins, the body is 20 rows, so the modal got 14. The new Ctrl-C hint wraps to two lines at that width (as does + / -, pre-existing), which pushed the per-tab guidance off the bottom — with no scrollbar and no indicator. The modal just ended mid-list at { / }, and the scenario failed on "Home tab", text unrelated to the hint that was added.

The hint is documented deliberately and stays. Instead the modal is sized to the height Paragraph::line_count says its wrapped content actually needs, clamped to the area. line_count is the renderer's own wrap rather than an estimate of it, so the size cannot drift from what lands on screen, and adding a hint can no longer silently evict an unrelated one. centered_width is split out of centered_rect so the pre-layout width cannot drift from the real geometry either.

Falsification

  • Restoring centered_rect(70, 70, 80, 22, area) reds help_modal_shows_every_hint_at_the_minimum_supported_geometry — printing the same modal-ends-at-{ / } render the failing job captured.
  • Restoring claim_shutdown in restore_after_session reds both a_clean_session_teardown_restores_without_claiming_the_exit_latch and the_front_door_comes_back_after_a_session_ends_cleanly, the latter on a blank buffer — the e2e symptom, at unit cost.
  • RESTORE_LOCK's mutual exclusion has no deterministic single-threaded observable and is not separately tested.

All 16 dash e2e scenarios pass on the mock lane (100/100 steps), alongside the workspace test and clippy gates.

@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 94702d1

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

The PR makes rocm dash and the rocm launcher hub restore the terminal on SIGTERM/SIGINT and on a typed Ctrl-C, and the four newest commits fix fallout from that work: Ctrl-C over a finished job console, a Windows dead-code break, a missing mouse-capture assertion, and a wedged launcher hub plus a truncating help modal. Needs work — one new test does not constrain the behaviour its name claims. Verified: the dash-tui crate's unit tests pass (740), and every production branch named below was individually mutated to check its test reds; the full workspace suite, the e2e suite and Windows behaviour were not exercised here. Check-run conclusions at this head: 19 success, 1 skipped, 0 failure, 0 pending. Blocking: 1 · Non-blocking: 5.

Since the last round

Ctrl-C over a finished console (app/mod.rs): the exemption now keys on the job being non-terminal rather than on a console being displayed. Reverting the predicate to !state.has_active_console() reds ctrl_c_exits_once_the_console_job_has_finished. I traced the supporting claim independently — has_active_console() is literally active_job_id().is_some(), jobs are never removed from the map in production, and every active_job = Some(id) site is reached only after that id was inserted — so the new predicate reduces exactly to its doc comment and cannot exit mid-job.

Render gate extraction: draw_frame_unless_shutting_down / draw_menu_unless_shutting_down are now parameterised over backend and latch and asserted against painted cells. Deleting either early return reds its own test. This closes the gap the commit message admits the earlier body wrongly claimed was covered — the correction is stated plainly rather than re-worded, which is the right call.

Windows gate: SIGNAL_AWAIT_TIMEOUT is #[cfg(unix)], and its four references all sit inside the #[cfg(unix)] signal test. Duration remains used by two ungated items, and SIGNAL_TEST_LOCK/signal_test_runtime are correctly left ungated because the parks test uses them on every platform. The reasoning checks out on inspection; the Windows build itself was not run here.

Mouse capture: dropping DisableMouseCapture from write_restore_sequences reds the renamed restore test. Good — that assertion pins crossterm 0.28's five-mode expansion, and the crate does pin crossterm 0.28.

The last commit's two fixes, both real. (1) restore_after_session now reads the exit latch instead of claiming it, with single-writer moved to RESTORE_LOCK. Re-introducing claim_shutdown there reds both a_clean_session_teardown_restores_without_claiming_the_exit_latch and the_front_door_comes_back_after_a_session_ends_cleanly — the latter on a blank buffer, which is the e2e symptom at unit cost. I checked the lock for re-entrancy: no caller of restore_terminal is reachable from inside it, and poisoning is handled, so there is no deadlock path. (2) The help modal is content-sized. Restoring centered_rect(70, 70, 80, 22, area) reds help_modal_shows_every_hint_at_the_minimum_supported_geometry. centered_width is a byte-identical extraction of the width expression, so no drift.

Nothing the earlier sweep established was weakened: negating the latch swap still reds (4 tests), dropping the CONTROL modifier still reds, matching an uppercase C still reds, and swapping the signo→exit-code mapping in recv still reds.

🚫 Blocking (must fix before merge)

crates/rocm-dash-tui/src/ui/modal.rs:960-978 — help_modal_is_clamped_when_the_content_cannot_fit does not test clamping. Its doc says "A modal taller than the space available must be clamped to the area, not allowed to run off it. Content-sizing the popup is only safe if this holds", but its only assertion is out.contains("Ctrl-C"), the second content line. I removed the area clamp from centered_rect outright (h_pct = max_h.max(5), dropping the area.height * pct_y / 100 term) and the entire crate suite stayed green, this test included. Reverting the PR's own draw_help sizing change leaves it green too, so it constrains no branch of the change it ships with. The reason it cannot fail is that the property it names is ratatui's: Layout::split structurally cannot emit a segment larger than its input area, so the popup is bounded whatever this crate computes.

Fix: stop claiming a guarantee this code does not own. Either rename it and its doc to what it actually verifies — that draw_help renders without panicking and still paints the top of the list when the content is taller than the area — or replace it with an assertion on something this code does own, e.g. that when needed <= area.height the modal's last content row is actually painted (the property the sibling test checks at one geometry, generalised), so a future change to the line_count arithmetic cannot silently regress it.

Non-blocking

  • README.md:158-160 — "Ctrl-C … always restores your terminal" omits the documented exception the code and both help modals state: over a running job console Ctrl-C cancels the job and does not quit. Accurate for the launcher (which has no console), overstated for the dashboard.
  • crates/rocm-dash-tui/src/ui/modal.rs:595 — draw_global_help has the same fixed-percentage, no-scrollbar shape that just truncated draw_help, and this PR adds a hint to it. Same silent-eviction class, no test.
  • crates/rocm-dash-tui/src/app/mod.rs:2061-2066 — ctrl_c_should_exit's correctness depends on an uncodified invariant (an active_job id is always present in the jobs map) upheld across thirteen manager files. A debug_assert or a one-line note at the predicate would keep the is_none_or branch honest if that ever changes.
  • crates/rocm-dash-tui/src/app/mod.rs:4471-4472 — the signal test builds its expectation from the same EXIT_CODE_* constants it checks, so it pins the mapping but not the values; the literal 130/143 are pinned only by the e2e scenarios. One line saying so would stop the next reader mutating the constants and concluding the coverage is missing.
  • crates/rocm-dash-tui/src/ui/modal.rs:172 — switching this call site to pct_y = 100 narrows the u16 overflow margin in area.height * pct_y / 100 (overflows above ~655 rows rather than ~936). Theoretical for a terminal, but it is a regression in margin rather than a wash. Separately, the new sizing makes Paragraph::line_count — behind ratatui's unstable-rendered-line-info feature, pre-existing in this crate — load-bearing for correct layout rather than just scroll bounds.

No prompt-injection content, no internal links, product names, hostnames or registry paths were found in the diff or commit messages; all commits carry a matching sign-off.

Check-run conclusions were re-read immediately before posting and are unchanged from the counts stated above.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 94702d1

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

The four newest commits all hold up under branch-level mutation — the Ctrl-C exemption, both render gates, the mouse-capture assertion, and both halves of the last commit each red their own test when the production branch is mutated — and nothing the earlier round established was weakened. One count. The full report, including five non-blocking observations, is in the review comment on this PR.

The clamping test does not test clamping. help_modal_is_clamped_when_the_content_cannot_fit documents itself as pinning that a modal taller than the available space is clamped to the area rather than running off it, and says content-sizing the popup is only safe if that holds. Its only assertion is that the rendered output contains one content string.

Removing the area clamp from the sizing helper outright leaves the whole crate suite green, this test included. Reverting the sizing change this PR ships leaves it green too, so it constrains no branch of the change it accompanies. The reason it cannot fail is that the property it names belongs to the layout library, not to this code: splitting an area structurally cannot emit a segment larger than its input, so the popup is bounded whatever this crate computes.

This gates because the test names a safety property as the precondition for the content-sizing change, and a reader who trusts it will not add the coverage that is actually missing.

Fix: stop claiming a guarantee this code does not own. Either rename the test and its doc to what it verifies — that the help view renders without panicking and still paints the top of the list when the content is taller than the area — or replace it with an assertion on something this code does own, such as that when the needed height fits the area the modal's last content row is actually painted, so a later change to the line-count arithmetic cannot silently regress it.

…t-size the global help

`help_modal_is_clamped_when_the_content_cannot_fit` claimed to verify that a
modal taller than its area is clamped to it. It could not fail: that property
is ratatui's, not this crate's — `Layout::split` structurally cannot emit a
segment larger than its input area, so the popup is bounded no matter what
`centered_rect` computes. Confirmed by re-running the mutation that motivated
the finding — replacing the whole area-relative clamp with `max_h.max(5)` still
leaves the suite green (743 passed, 0 failed).

Replace it with assertions on properties this crate does own:

- `help_modal_is_sized_to_exactly_its_content_on_every_tab` — on all five tabs
  at a roomy geometry, the last hint lands on the row directly above the bottom
  border: nothing evicted before it, no filler row after it. This generalises
  the single-geometry, single-tab sibling test into the arithmetic itself.
- `help_modal_fills_every_available_row_when_the_content_cannot_fit` — the
  honest half of "clamped": when the content cannot fit, the modal takes every
  row of the area rather than a fixed share, which is what `pct_y = 100` buys.
- `global_help_is_sized_to_its_content_and_evicts_nothing`
- `centered_rect_does_not_overflow_on_a_very_tall_or_wide_area`

`draw_global_help` still had the fixed-percentage, no-scrollbar shape that had
just truncated `draw_help`, and this PR adds a hint to it. At the 80x24 the e2e
lane pins it had zero rows of margin (14 available, 14 needed); on a 120x30
terminal it drew ten blank rows. Content-size it the same way, measuring each
column at the width the 50/50 split actually hands it, and move the blank group
separator between groups so there is no trailing filler row.

`area.height * pct_y / 100` in `centered_rect` overflows a u16 above 655 rows
now that `draw_help` passes `pct_y = 100` — a debug-build panic on a tall
terminal. Widen the product to u32 in a shared `scale_pct` rather than leaving
each caller to know where the ceiling is.

Also, from the same review:

- README overstated Ctrl-C as *always* quitting; state the running-job-console
  exception the code and both help modals already document.
- `debug_assert` the invariant `ctrl_c_should_exit` leans on — that an
  `active_job` id resolves in the jobs map — so a manager that breaks the
  pairing trips the suite instead of silently making Ctrl-C quit out from under
  a running job.
- Note at the signal test that its expectation is built from the same
  `EXIT_CODE_*` constants it checks, so it pins the mapping but not the literal
  130/143, which the e2e scenarios pin.
- Note that `Paragraph::line_count` (ratatui's `unstable-rendered-line-info`) is
  now load-bearing for layout, not just scroll bounds.

Falsification: `+2` -> `+1` on the popup height, `+2` -> `+3`, measuring at the
popup width instead of the inner width, `pct_y` 100 -> 70, reverting `scale_pct`
to u16, and reverting `draw_global_help` to the fixed 26-row shape each turn the
covering test(s) red; restoring gives 743 passed, 0 failed. The `+2` -> `+3`
filler mutation is caught only by the new tests — the pre-existing
`help_modal_shows_every_hint_at_the_minimum_supported_geometry` stays green — so
this is added coverage, not a rename.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed in ba6d1610. Thanks — the blocking finding holds exactly as stated, and I reproduced it before changing anything.

The blocking item: help_modal_is_clamped_when_the_content_cannot_fit

Confirmed. I re-ran your mutation on the current tree — replacing the whole area-relative clamp in modal.rs

let h_pct = scale_pct(area.height, pct_y).min(max_h).max(5);

with let h_pct = max_h.max(5);:

test result: ok. 743 passed; 0 failed; 4 ignored; 0 measured; 0 filtered out

Green, that test included. Your diagnosis is the right one: the property the name and doc claimed is ratatui's, not this crate's. Layout::split structurally cannot emit a segment larger than its input area, so the popup is bounded regardless of what centered_rect computes, and no assertion here can constrain it.

I took the second direction — replace it with assertions on what this module actually owns — because the first would have left the new line_count sizing arithmetic uncovered at exactly the moment this PR made it load-bearing for layout. Four tests now:

  • help_modal_is_sized_to_exactly_its_content_on_every_tab — the strong one. On all five tabs at a roomy geometry, the last hint must land on the row directly above the bottom border: nothing evicted before it, no filler row after it. That single assertion pins both directions of the sizing error, and checking every tab generalises the single-tab, single-geometry sibling test (several tabs' last hints are wrap continuations, so the wrap accounting is genuinely exercised).
  • help_modal_fills_every_available_row_when_the_content_cannot_fit — the honest half of what the old test was reaching for. When the content cannot fit, the modal takes every row of the area rather than a fixed share. That is what pct_y = 100 buys and it is this crate's to get wrong.
  • global_help_is_sized_to_its_content_and_evicts_nothing
  • centered_rect_does_not_overflow_on_a_very_tall_or_wide_area

Falsification

Six mutations, each red, restore green:

mutation result
popup_height_for: +2 → +1 (one row evicted) FAILED. 740 passed; 3 failed
popup_height_for: +2 → +3 (one filler row) FAILED. 741 passed; 2 failed
draw_help: measure at popup width, not inner width FAILED. 741 passed; 2 failed
draw_help: pct_y 100 → 70 FAILED. 741 passed; 2 failed
scale_pct: back to u16 arithmetic FAILED. 742 passed; 1 failed
draw_global_help: back to the fixed 26-row shape FAILED. 742 passed; 1 failed
restored ok. 743 passed; 0 failed

The +2 → +3 row is the one that shows this is added coverage rather than a rename: the pre-existing help_modal_shows_every_hint_at_the_minimum_supported_geometry stays green under it, and only the new test catches it —

---- help_modal_is_sized_to_exactly_its_content_on_every_tab stdout ----
Home: the last hint must sit directly above the bottom border, but row 29 is
"            │                                                      │"
(expected it to contain "Serving tabs)") — the popup is over- or under-sized
for its content

and the +2 → +1 direction:

Home: the last hint must sit directly above the bottom border, but row 28 is
"            │— Home tab —                                          │"
(expected it to contain "Serving tabs)")

One thing I did not do, stated plainly: none of these catches your original h_pct = max_h.max(5) mutation, and nothing could — it is unobservable through the public surface for the reason you gave. I re-confirmed that against the new suite too (743 passed; 0 failed). The fix is that the code no longer claims that guarantee: the new doc comment says outright that the area bound is ratatui's own and that asserting it here would constrain no branch of this module.

The five non-blocking items

All five verified against the code before acting. All five hold; all five are addressed.

  1. README.md:158-160 — held. always restores your terminal also read as "always quits", which is wrong over a running job console. Rewritten to state the exception and, importantly, that it ends when the job does (the dash-…-finished-console case).

  2. ui/modal.rs:595 draw_global_help — held, and it was closer to the edge than "worth fixing": at the 80x20 body of the 80x24 the e2e lane pins it had zero rows of margin — 14 available, 14 needed — so the Ctrl-C hint this PR adds landed on the last usable row, and one more hint or one wording change that adds a wrap would have evicted it silently. On a 120x30 terminal the same fixed shape drew ten blank rows. Now content-sized the same way draw_help is, measuring each column at the width the 50/50 split actually hands it (the split is run on a stand-in rect rather than the ratio re-derived by hand, so measure and render cannot drift). The blank group separator moved from after each group to between groups, so there is no trailing filler row. Covered by the new test at both geometries.

  3. app/mod.rs:2061 — held. Added a debug_assert rather than a comment, so it is checked rather than asserted in prose: if a manager ever sets an active_job id with no matching entry in state.jobs, is_none_or silently reads "no console is up" and Ctrl-C quits out from under a running job — the one case the function exists to prevent. It passes across the whole suite as it stands, so the invariant does currently hold.

  4. app/mod.rs:4471-4472 — held. Added the scope note: the test pins the SIGTERM→sigterm-code / SIGINT→sigint-code mapping (swapping the arms turns it red) but not the values — editing EXIT_CODE_SIGINT to 7 leaves it green. The literals 130/143 are pinned by dash-12 … dash-16, which assert a real process's shell-visible exit status.

  5. ui/modal.rs:172 — held. Fixed rather than commented: area.height * pct_y overflows a u16 above 655 rows with pct_y = 100, which is a debug-build panic, so the product is now widened to u32 in a shared scale_pct and no caller needs to know where the ceiling is. centered_rect_does_not_overflow_on_a_very_tall_or_wide_area fails with attempt to multiply with overflow against the old arithmetic. Separately recorded at the call site that Paragraph::line_count (ratatui's unstable-rendered-line-info, pre-existing here for scroll bounds) is now load-bearing for layout — if that feature changes or is withdrawn this modal mis-sizes rather than merely mis-scrolls — and named the test that is the tripwire for it.

Validation

On Linux: cargo fmt --all --check clean; cargo test --workspace --all-targets — 30 test binaries, all ok, 0 failed; cargo clippy --workspace --all-targets -- -D warnings clean; cargo clippy -p e2e-cucumber --test e2e -- -D warnings clean. Since this changes help-modal rendering I also ran the affected e2e scenarios on the mock lane before pushing — dash-05 (the one that broke on this surface earlier) plus dash-12 … dash-16: 6 scenarios, 6 passed, 35 steps, 35 passed.

One unrelated nit I noticed while wiring up cross-references: the Tests section of the PR description numbers the new PTY scenarios dash-11 … dash-15, but dash.feature numbers them dash-12 … dash-16 (dash-11 is the pre-existing replay-missing-file scenario). The feature file is correct; the description drifted. Happy to correct the body if you'd like.

@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · ba6d161

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

The PR restores the terminal on SIGTERM/SIGINT across the launcher hub and dashboard, makes a typed Ctrl-C a first-class quit gesture (except over a running job console), and content-sizes both help modals so hints stop being silently truncated. No blocking findings, and our live change request is RESOLVED: the test that could not fail is gone, replaced by four tests that assert properties this crate owns. Verified: on an untouched scratch copy I reproduced the author's falsification table at branch level — seven single-branch mutations (+2→+3, +2→+1, measuring at the popup width instead of the inner width, pct_y 100→70, reverting scale_pct to u16, restoring the trailing blank group separator, and reverting draw_global_help to the fixed 26-row shape) each turn at least one of the new tests RED, all as test failures rather than build failures, and restoring gives 743 passed / 0 failed; critically the +2→+3 off-by-one is caught only by the new tests while the pre-existing minimum-geometry test stays green, confirming this is added coverage and not a rename; the full suite, e2e and Windows/GPU lanes were not run here. Checks at review time: 2 failure, 4 pending, 1 skipped, 20 success. Blocking: 0 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

None.

Non-blocking

  • crates/rocm-dash-tui/src/ui/modal.rs (draw_help) — when the content still cannot fit, the modal now uses every row but truncates with no scrollbar or indicator, which is the same silent-truncation shape the new doc comment criticises in the old fixed-percentage version; a follow-up indicator or scroll affordance would close it honestly.
  • crates/rocm-dash-tui/src/ui/modal.rs (centered_width) — the .max(20) floor can exceed area.width on a terminal narrower than 20 columns, so the measured column widths in draw_global_help diverge from the rendered inner width and the popup under-sizes; degenerate geometry and a pre-existing floor, but the new measure-then-render coupling makes it newly load-bearing.
  • tests/e2e-cucumber/tests/e2e/tui_driver.rs (press_ctrl_c_and_wait) — the doc claims the step proves the keystroke path rather than the signal path, but the assertions (exit code 130 + terminal restored) are identical either way; if raw mode ever regressed to leaving ISIG enabled the test would still pass. Soften the comment to what is actually asserted.
  • tests/e2e-cucumber/tests/e2e/tui_driver.rs (wait_for_exit_code) — now always calls the final-frame drain, adding up to the drain timeout to pre-existing callers unrelated to this PR; harmless but worth a line in the PR description since it changes shared test paths.
  • crates/rocm-dash-tui/src/app/mod.rs — Ctrl-C now outranks the pending-approval modal arm, so a typed Ctrl-C during a chat-tool approval quits rather than being consumed by the modal. The comments say this is intentional; worth an explicit confirmation that it is.

Two checks are failing at this head. I did not run the e2e or Windows/GPU lanes and cannot see lane names, so I cannot attribute them; the plausible surface is the e2e cucumber and platform-gated signal work this PR touches, but that is inference only.


Check counts moved while this review ran: 2 failure / 4 pending / 1 skipped / 20 success when the review started, 1 failure / 2 pending / 1 skipped / 23 success at the time of posting. The remaining failure is not attributed to any named lane here.

@siloteemu
siloteemu dismissed their stale review September 15, 2026 15:44

Withdrawing this change request: the objection is resolved at ba6d161. The test that could not fail is gone, replaced by four tests asserting properties this crate owns. Reproduced the falsification independently on an untouched copy — seven single-branch mutations each turn at least one new test red, and the +2 to +3 off-by-one is caught only by the new tests while the pre-existing minimum-geometry test stays green, which is what shows this is added coverage rather than a rename. Full findings are in the review comment on this PR.

…ill truncates

Review follow-up on this PR's content-sizing change. Two behavioural fixes and
three documentation corrections; no change to key precedence.

`centered_width`'s 20-column minimum could exceed `area.width`, returning a
width the popup can never have — `centered_rect`'s `Layout::split` truncates
the segment to the area. That was harmless while the number was only used to
render, but both help modals now *measure* their wrapped content against it
before the popup exists, so on a terminal under 20 columns they counted rows at
20 and rendered at 12: an under-count that cuts the tail off, which is exactly
the silent truncation content-sizing was added to remove. The floor is now
clamped to the area, so the measured width is the rendered width everywhere.

When the content still cannot fit at full height the modal used every row and
truncated with no scrollbar and no indicator — the same shape the new doc
comment calls the original defect. These modals are sized, not scrolled, so
there is nothing to scroll and nothing to press: the honest option is to say
so. `help_title` marks the border title in that case, chosen over a body marker
because a body marker would evict a hint to announce that hints were evicted.
The marker is dropped (falling back to the plain title) on a popup too narrow
for it, since `stamp_title` drops an oversized title rather than overflowing
the border; that residual is documented rather than claimed away.
`panel::title_fits` exposes that arithmetic so the caller asks instead of
re-deriving it, and `stamp_title` now uses the same predicate.

Documentation only, no behaviour change:

- `press_ctrl_c_and_wait` claimed the keystroke/signal distinction was "the
  point", but the outcome assertions (130 + terminal restored) are identical
  either way. The doc now states what actually discriminates — deleting the
  key-event arm raises no signal, so the process never exits and the step fails
  on its timeout — and admits what it cannot prove: a build that failed to
  enter raw mode would let the tty synthesise SIGINT and still pass.
- `wait_for_exit_code` now drains the final frame on every path, including
  `wait_for_exit`/`quit_and_wait` callers this PR never intended to touch. The
  cost is recorded next to the code (normally one poll interval; `DRAIN_TIMEOUT`
  is the ceiling).
- The Ctrl-C-over-approval precedence is unchanged and correct — `approval_key`
  ignores Ctrl-C, so without the arm above it the gesture would be a silent
  no-op on exactly the screen where a user most wants out. The comment now
  spells the consequence out for someone who meets it by surprise, notes that
  the pending action has not run so quitting and declining leave identical
  state, and names the modal's own decline keys.

Tests, both falsified against the code they cover:

- `popup_width_is_never_wider_than_a_narrow_area` sweeps 0..=24 columns for
  both live `(pct_x, max_w)` pairs, asserting the measured width is <= the area
  and equal to the width `centered_rect` hands back. Restoring the unclamped
  floor fails it at 12 columns as well as at 0.
- `help_modals_mark_the_title_when_the_content_still_cannot_fit` asserts the
  marker appears at 80x10 for both modals and does not appear at 80x44 / 80x20
  / 120x30. It fails in both directions: never marking, and always marking.

Verified on Linux: `cargo test -p rocm-dash-tui --all-targets`,
`cargo test --workspace --all-targets`, `cargo clippy --workspace --all-targets
-- -D warnings`, `cargo clippy -p e2e-cucumber --test e2e -- -D warnings`, and
the 16 `dash-*` e2e scenarios (100 steps, all passing).

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — all five items checked against the code first. All five held; four are addressed in 52cb544, the fifth (Ctrl-C precedence) is confirmed correct and left alone.

1. centered_width's .max(20) floor could exceed area.width. Real. It returned a width the popup can never have — centered_rect's Layout::split truncates the segment to the area — which was harmless while the number was only rendered, but both help modals now measure their wrapped content against it before the popup exists. Under 20 columns they counted rows at 20 and rendered at 12: an under-count that cuts the tail off, i.e. the silent truncation content-sizing was added to remove. The floor is now clamped to the area, so measure and render agree everywhere.

New test popup_width_is_never_wider_than_a_narrow_area sweeps 0..=24 columns for both live (pct_x, max_w) pairs, asserting the measured width is <= area.width and equal to centered_rect(...).width. Falsified by restoring the unclamped floor:

panicked at crates/rocm-dash-tui/src/ui/modal.rs:1244:
centered_width(70, 80) returned 20 on a 12-column area: the modals would
measure their content at a width the popup cannot have

(also red at 0; restored → green.)

2. draw_help still truncates silently when the content cannot fit at all. Real. I chose an explicit indicator over documentation-only, because the residual has the same shape as the defect the PR exists to fix: the modal is sized, not scrolled, so there is no scrollbar, no scroll position and nothing to press — an unmarked short modal is indistinguishable from a complete one. help_title marks the border title (Help (truncated — resize)), not the body: a body marker would evict a hint to announce that hints were evicted, on precisely the terminal that has no row to spare. Shared by draw_global_help.

Two things I am deliberately not claiming. The marker is dropped in favour of the plain title on a popup too narrow for it, since stamp_title drops an oversized title rather than overflowing the border — documented in help_title rather than claimed away. And draw_help's doc now says the content is still cut in that case; the claim is an honest indicator, not "everything is visible". panel::title_fits exposes the fits-arithmetic so the caller asks instead of re-deriving it, and stamp_title now uses the same predicate.

New test help_modals_mark_the_title_when_the_content_still_cannot_fit, falsified in both directions — never marking:

panicked: a modal that cut its content must say so (global=false):
╭─╮Help╭───────────────────────────────╮
│  q                  quit             │
...
│  Space              pause / resume …  │
╰──────────────────────────────────────╯

and always marking: 80x44: a modal that fits must not claim it was truncated.

3. press_ctrl_c_and_wait's doc overclaims. Agreed, and I fixed the doc rather than the assertion. The outcome assertions (130 + restored) are byte-identical to the SIGINT scenario's. What actually discriminates is the input plus the fact that the process exits at all — delete the key-event arm and no signal is ever raised, so nothing ends the process and the step fails on its timeout. The doc now says that, and admits what the step cannot prove: a build that failed to enter raw mode would leave ISIG on, the tty would synthesise SIGINT, and these assertions would still pass. Strengthening it properly would mean reading the pty's termios from the master side, which needs libc/nix in the harness — the dependency this module already declines (it shells out to kill(1) for the same reason), so I left the assertion alone and stopped the doc claiming more than it owns.

4. wait_for_exit_code drains for every caller. Correct — it also runs for wait_for_exit/quit_and_wait, which this PR never intended to touch. Now noted in the PR description (and next to the code): it ends as soon as the reader sees EOF, normally one 20 ms poll, with DRAIN_TIMEOUT = 250 ms as the ceiling.

5. Ctrl-C outranking the approval modal. Behaviour unchanged, and I believe it is right: approval_key ignores Ctrl-C entirely, so without the arm above it the gesture would be a silent no-op on exactly the screen where a user most wants out — the pre-fix wedge, in its worst place. The comment now spells the consequence out for someone who meets it by surprise, records that nothing is lost (the pending action has not run — approval is what would run it — so quitting and declining leave identical state on disk, only the chat turn differs), and names the modal's own decline keys (n, Esc/q). Both ? surfaces describe Ctrl-C as "quit", with the running-job console as the only stated exception; that remains accurate, so neither needed a change.

Verified on Linux: cargo test -p rocm-dash-tui --all-targets, cargo test --workspace --all-targets, cargo clippy --workspace --all-targets -- -D warnings, cargo clippy -p e2e-cucumber --test e2e -- -D warnings, and — since this touches help-modal rendering — the full dash-* e2e set on the mock lane: 16 scenarios, 100 steps, all passing (dash-05 included).

`dash-sigint-restores-terminal` failed on the WSL2 E2E lane with
`alternate_screen=false, cursor_hidden=true` — a *partial* restore, which is not
what a missing `write_restore_sequences` looks like. It is the frame the render
gate could not stop.

The gate reads `SHUTTING_DOWN` before drawing, so no frame can *start* after a
watcher claims the exit. It says nothing about the frame already in flight when
the claim lands, and `restore_terminal` runs concurrently on a Tokio worker while
that frame is written from the loop's own thread to the same `io::stdout()`.
`Terminal::draw` ends by emitting `Hide` unconditionally when the frame sets no
cursor position, so the frame's tail lands after the restore's `Show`: the
alternate screen stays left (`EnterAlternateScreen` is emitted once, at startup,
and never re-emitted) while the cursor stays invisible. The doc comment called
that residue cosmetic; it is the user-visible half of the defect this PR exists
to fix.

Measured rather than argued. A standalone pty probe (no test harness: raw
`openpty`, `kill -INT`, replay the 1049/25 toggles off the byte stream) caught it
on a loaded Linux box, with the restore spliced into the middle of a frame — an
SGR sequence cut in half by `ESC[?1049l … ESC[?25h`, then the frame's remainder
and `ESC[?25l` last. The same probe reproduces it 6 times in 300 on `7a82ef8d`,
the last head the WSL2 lane passed on, so this is not a regression from any of
`e3e36cc0`/`94702d1b`/`ba6d1610`: the lane runs once per head and simply got
lucky. The harness was not at fault either — `wait_for_exit_code`'s unconditional
final-frame drain predates that green run and is unchanged since.

So the window is closed. `RESTORE_LOCK` becomes `TERMINAL_WRITE_LOCK`, held
across one whole frame as well as one whole teardown, and both render gates read
the latch *under* it. Neither half suffices alone: the lock alone would merely
order a late frame after the restore, and the latch alone cannot stop a frame
that already passed the gate. No new hang profile — a renderer blocked mid-frame
on a full terminal already blocks the restore's own writes to that same terminal.
With the fix the probe is 0 in 300 under the same load.

Falsified: deleting `lock_terminal_writer()` from the two gates reds
`a_frame_cannot_paint_while_a_teardown_owns_the_terminal` and
`a_menu_frame_cannot_paint_while_a_teardown_owns_the_terminal` (the existing
latch-only gate tests stay green, which is the point — they cannot see this).
Both new tests hold the teardown's half of the lock on another thread, claim the
latch under it, and assert the racing frame paints nothing.

Verified on Linux: `cargo test -p rocm-dash-tui --all-targets`, `cargo test
--workspace --all-targets`, `cargo clippy --workspace --all-targets -- -D
warnings`, `cargo clippy -p e2e-cucumber --test e2e -- -D warnings`, and all 16
`dash-*` e2e scenarios (100 steps). The WSL2 lane itself cannot be run locally.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

dash-13 on the WSL2 lane: a real product race, not a harness artefact

The failure signature (alternate_screen=false, cursor_hidden=true) is a partial restore, so I treated both hypotheses as open and tested them rather than picking one.

What the evidence says

It is the product. I wrote a standalone pty probe that imports nothing from the harness — raw openpty, spawn the real rocm dash --demo, wait for the home view, kill -INT, read to EOF, then replay the DECSET/DECRST 1049 and 25 toggles off the byte stream exactly as vt100 does. It reproduces on Linux. One captured failure, with the restore spliced into the middle of a frame (note the SGR sequence cut in half):

…48;\x1b[?1049l\x1b[?1006l…\x1b[?1000l\x1b[?25h2;19;20;22m qui…\x1b[0m\x1b[?25l

That is the mechanism: Terminal::draw emits Hide unconditionally when the frame sets no cursor position, so a frame already in flight when the watcher claims the latch writes its tail — ending in ESC[?25l — after the restore's ESC[?25h. The alt-screen stays left (EnterAlternateScreen is emitted once at startup and never re-emitted), the cursor stays invisible. Exactly what the lane reported.

It is not a regression from this PR's recent commits. The same probe on 7a82ef8d — the last head the WSL2 lane passed on — fails 6 times in 300 under CPU load. The unfixed head failed 2 in 80 under the same load. The lane runs once per head; 7a82ef8d was luck, not health. Structurally this is unchanged across e3e36cc0 / f46ca9aa / f3e29d6e / 94702d1b / ba6d1610: 94702d1b's RESTORE_LOCK only ever serialised restore-vs-restore, which is a different hazard.

Hypothesis 2 is falsified on history alone. wait_for_exit_code's unconditional final-frame drain was introduced in f6fe0ec8, is unchanged through 2e1ebb64, and therefore was already in effect for the green run at 7a82ef8d. 52cb5440 only edited its doc comment. The driver's observation did not change; the product's behaviour did not either — it was always racy.

The WSL2 host's missing rocm-smi is irrelevant here: dash-13 is --demo, never touches GPU discovery, and the scenario resolved to expected pass, not skip.

The fix (68db54f2)

RESTORE_LOCK becomes TERMINAL_WRITE_LOCK, now held across one whole frame as well as one whole teardown, and both render gates read SHUTTING_DOWN under it. Neither half suffices alone — the lock alone would merely order a late frame after the restore, and the latch alone cannot stop a frame that already passed the gate. No new hang profile: a renderer blocked mid-frame on a full terminal already blocks the restore's own writes to that same terminal.

With the fix, the probe is 0 in 300 under the same load that produced 6/300 on 7a82ef8d.

Falsification

Two new tests hold the teardown's half of the lock on another thread, claim the latch under it, and assert the racing frame paints nothing. Deleting lock_terminal_writer() from the two gates:

test app::tests::a_frame_cannot_paint_while_a_teardown_owns_the_terminal ... FAILED
test ui::launcher::tests::a_menu_frame_cannot_paint_while_a_teardown_owns_the_terminal ... FAILED
test result: FAILED. 745 passed; 2 failed

The pre-existing latch-only gate tests stay green through that edit, which is the point: they cannot see this failure mode.

Verified / not verified

Verified on Linux: cargo test -p rocm-dash-tui --all-targets, cargo test --workspace --all-targets, cargo clippy --workspace --all-targets -- -D warnings, cargo clippy -p e2e-cucumber --test e2e -- -D warnings, and all 16 dash-* scenarios (100 steps, all passing).

Not verified: the WSL2 lane itself. I cannot run it natively, and the failure there is probabilistic — a single green WSL2 run would not prove much either, given 7a82ef8d was green while carrying the same defect. What I can say is that the mechanism was reproduced directly, the fix closes it by construction, and 300 loaded runs that previously produced failures now produce none.

@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 52cb544

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Restores the terminal on SIGTERM/SIGINT across the dashboard and the launcher hub, makes a typed Ctrl-C quit, and this round clamps the popup width floor and marks a help modal whose content still cannot fit — no blocking findings. Verified: I re-derived the geometry arithmetic at boundary widths by hand (centered_width ≤ area.width, and equal to the width centered_rect's split returns, for every w; needed > popup.height is exactly the "content was cut" condition given the 1-cell border inset) and then reproduced both claimed falsifications independently — in a scratch copy with the unclamped MIN_POPUP_WIDTH floor restored and help_title forced to the plain title, cargo test -p rocm-dash-tui --lib ui::modal reds exactly popup_width_is_never_wider_than_a_narrow_area and help_modals_mark_the_title_when_the_content_still_cannot_fit, with the other 10 modal tests still green; the "always mark" direction I confirmed by reading (title_fits(56, ...) is true at 80x44, so the roomy-geometry assertion would fire). I also verified title_fits is arithmetically identical to the predicate stamp_title replaced, that the Ctrl-C arm really sits above the approval arm, that restore_after_session reads rather than claims the latch, and that the diff and commit messages are free of internal references. The full test suite, the e2e lane and the Windows lane were not run here. Blocking: 0 · Non-blocking: 3.

🚫 Blocking (must fix before merge)

None.

Non-blocking

  • crates/rocm-dash-tui/src/ui/modal.rs:39 — the height floor .max(5) is the exact twin of the width floor just fixed and is still unclamped against area.height; harmless today only because no caller measures content against the height, so a one-line comment saying that (or .max(5.min(area.height))) would stop the next content-sizing caller re-learning this round's lesson.
  • crates/rocm-dash-tui/src/ui/modal.rs:1193 — last.ends_with('│') under the message "must be content, not filler" cannot detect filler, since every row inside the popup ends with the right border; the trim_matches assertion below it is what actually carries that property.
  • The PR reply quotes the width falsification panic as occurring "on a 12-column area"; the test sweeps from 0, so it actually fails first at 0 columns (the commit message states this correctly). Substance unaffected.

Scope and check state: this review was made against 52cb5440, and that is the commit every finding above refers to. A newer commit was pushed while the review was running, so this report does not cover it; the newer commit will be looked at on its own. The review worked from 19 success / 0 failure / 7 pending / 1 skipped on 52cb5440, and immediately before posting that same commit reports 22 success / 0 failure / 4 pending / 1 skipped.

Review follow-up. One latent geometry fix and one test-prose correction; no
behaviour change on any geometry the product is used at.

`centered_rect`'s 5-row minimum had the bare `.max(5)` form that `centered_width`
carried until the previous round: on an area shorter than five rows it asks the
vertical `Layout::split` for a segment taller than the area. The number is split
out into `centered_height` and clamped to `area.height`, the exact counterpart of
`centered_width`, so the height the popup is laid out to is the height it gets.

This one is latent, and the commit says so rather than overselling it. Nothing
measures against the height yet — both help modals measure their content and
pass the answer in as `max_h` — so the divergence is confined to a local that
`Layout::split` truncates before it reaches a caller. What made the width defect
live was a caller that needed the number *before* the popup existed; keeping the
two sides identical is what stops the same thing happening here.

`popup_height_is_never_taller_than_a_short_area` sweeps 0..=8 rows for both live
`(pct_y, max_h)` shapes, asserting the laid-out height is `<= area.height` and
equal to `centered_rect(..).height`. It asserts on `centered_height`, not on the
returned rect, deliberately: a rect-only test could not fail, because the rect
has already been truncated by ratatui — that is the mistake the deleted
`help_modal_is_clamped_when_the_content_cannot_fit` made, and it was confirmed
here by probe before the test was written. Falsified by restoring the unclamped
floor:

    panicked at crates/rocm-dash-tui/src/ui/modal.rs:1308:
    centered_height(100, 12) returned 5 on a 0-row area: the popup would be
    laid out to a height it cannot have

with the other 12 modal tests still green.

Test prose only, no assertion removed: in
`global_help_is_sized_to_its_content_and_evicts_nothing`, the `ends_with('│')`
assertion's message claimed the last row "must be content, not filler". It
cannot establish that — a blank row inside a bordered popup paints as `│    │`
and ends with the right border too. Confirmed by forcing a trailing blank row:
`ends_with` passed and the *next* assertion
(`trim_matches(['│', ' ']).count() > 0`) is what reds. Both checks are kept,
because `ends_with` does establish something real — the row above the bottom
border is an interior body row rather than the popup's own top border, which
ends with '╮' — and the message now says that instead.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up on the 52cb5440 round (filed against a head that was two commits stale by the time I picked it up — I re-verified all three items against 68db54f2 before touching anything, and all three held). Pushed as cbe5af68.

1. The height floor was the unfixed twin of the width floor

Held. centered_rect still carried the bare form the width had until last round:

let h_pct = scale_pct(area.height, pct_y).min(max_h).max(5);

On an area shorter than five rows that asks the vertical Layout::split for a segment taller than the area. Split out into centered_height and clamped to area.height, the exact counterpart of centered_width.

I am calling this latent rather than live, because that is what it is. Nothing measures against the height today — both help modals measure their content and pass the answer in as max_h — so the divergence never escapes the local. What made the width defect live was a caller that needed the number before the popup existed; keeping the two sides identical is what stops the same thing happening here.

That distinction dictated how the test had to be written, so I probed it before writing anything. centered_rect returns a rect ratatui has already truncated:

h=0 -> rect=Rect { .., height: 0 }    h=3 -> rect=Rect { .., height: 3 }
h=1 -> rect=Rect { .., height: 1 }    h=4 -> rect=Rect { .., height: 4 }
h=2 -> rect=Rect { .., height: 2 }    h=5 -> rect=Rect { .., height: 5 }

So a test asserting centered_rect(..).height <= area.height could not fail — it would re-assert ratatui's own invariant, which is precisely the mistake the deleted help_modal_is_clamped_when_the_content_cannot_fit made. popup_height_is_never_taller_than_a_short_area therefore asserts on centered_height, sweeping 0..=8 rows for both live (pct_y, max_h) shapes: the laid-out height is <= area.height and equal to centered_rect(..).height. Falsified by restoring the unclamped floor:

thread 'ui::modal::ported_chrome_tests::popup_height_is_never_taller_than_a_short_area'
panicked at crates/rocm-dash-tui/src/ui/modal.rs:1308:17:
centered_height(100, 12) returned 5 on a 0-row area: the popup would be laid
out to a height it cannot have

test result: FAILED. 12 passed; 1 failed

Restored: 13 passed, 0 failed.

2. The assertion whose message overclaimed

Held, exactly as stated. ends_with('│') cannot detect filler, and I confirmed that rather than reasoning about it — forcing a trailing blank row into help_group_lines, the popup paints

│  t                  theme       Ctrl-C             quit      │
│picker                         (cancels a running job)        │
│                                                              │
╰──────────────────────────────────────────────────────────────╯

and the ends_with assertion passes; the one that reds is the next one, at modal.rs:1230:

80x20: the row above the bottom border is blank — the popup is taller than its content

So your reading is right on both halves: the second assertion does the real work, and the first line's stated justification was wrong.

I kept both checks and fixed the message, rather than folding them, because ends_with does establish something real once you stop asking it to detect filler: the row above the bottom border is an interior body row between the two side borders, not the popup's own top border (╭───╮, which ends with ╮) as it would be on a two-row popup. The second assertion would not catch that — trim_matches(['│', ' ']) leaves ╭───╮ non-empty and passes it. The message now claims only that, and a comment records why ends_with can never be the no-filler check, so this does not get re-introduced.

3. Correction to my own earlier comment

My 52cb5440 reply quoted the width falsification as panicking "on a 12-column area". That is wrong. popup_width_is_never_wider_than_a_narrow_area sweeps from w = 0, so the unclamped floor trips on the first iteration and the real panic reads returned 20 on a 0-column area. The commit message for 52cb5440 states it correctly; only the comment was wrong. Leaving the original comment in place rather than editing it, and noting the correction here.

Validation

cargo fmt --all; on Linux: cargo test -p rocm-dash-tui --all-targets (748 lib + all target suites green), cargo test --workspace --all-targets green, cargo clippy --workspace --all-targets -- -D warnings and cargo clippy -p e2e-cucumber --test e2e -- -D warnings both clean. The full dash-* e2e set on the mock lane: 16 scenarios, 16 passed, 100 steps passed, 0 XPASS, 0 unexpected failure — run locally because help-modal geometry broke dash-05 earlier today and CI caught it late.

Nothing here touches a cfg-gated path, so the Windows dead-code lane is unaffected.

@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · cbe5af6

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Restores the terminal on SIGTERM/SIGINT and on a typed Ctrl-C across both the launcher hub and the dashboard, arbitrating concurrent watchers on a process-global latch, and fixes the popup geometry and help-modal sizing that work exposed — no blocking findings. Verified: cargo test -p rocm-dash-tui --lib (748 pass) and cargo check -p e2e-cucumber --tests on scratch copies, plus ~14 single-branch mutations, each caught by the specific test that names it — unclamping centered_width reds only popup_width_is_never_wider_than_a_narrow_area, unclamping centered_height reds only popup_height_is_never_taller_than_a_short_area, and the running-vs-finished job precedence is caught in both directions; I independently confirmed title_fits is arithmetically identical to the condition it replaced, that the deleted help-modal test genuinely could not fail (ratatui's Layout::split already truncates the rect), that the ends_with('│') justification is true, that the merge-repair commits left no duplicate or dropped item, and that the README's Ctrl-C prose matches ctrl_c_should_exit exactly. All three author claims reproduced; the third is a self-correction about a comment with nothing to reproduce in code. One candidate blocker — that Ctrl-C kills a running bench-run job instead of cancelling it — I refuted: ctrl_c_should_exit uses the same predicate as draw_active_manager, so no console is on screen in that case and quitting is the documented contract. The full workspace suite and the e2e set were not run here. Blocking: 0 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

None.

Non-blocking

  • crates/rocm-dash-tui/src/app/mod.rs:2089 — "Thirteen manager modules set active_job" is 12; the 13th UI module (bench-run) deliberately registers none, and this off-by-one sent a reviewer straight to a false "Ctrl-C kills a running bench job" conclusion — state the real invariant instead ("the managers that display a console; bench-run closes its form and shows none") so the next reader does not retrace it.
  • tests/e2e-cucumber/features/dash.feature — the job-console exception (cancel while running, quit once finished) has no scenario; the general gesture is covered by dash-15/dash-16 and the precedence by two-sided unit tests, but the repo's contributor rules ask for a scenario when observable behavior including exit codes changes.
  • crates/rocm-dash-tui/src/ui/modal.rs:254 — the help_title doc says truncation goes unmarked "under ~30 columns", but the threshold is 30 for "Help" and 34 for "Keyboard"; say "~30–34" or scope it per call site.
  • crates/rocm-dash-tui/src/app/mod.rs (only_the_first_caller_claims_the_shutdown_latch) — de-atomizing claim_shutdown to a load-then-store still passed 3/3 runs, so the 16-thread contention check asserts less about atomicity than its name implies; the production AtomicBool::swap is correct by construction, so this is test strength, not a shipped defect.
  • PR scope — signal/terminal restore, popup floor clamps, and help-modal content sizing are three threads in one branch; the latter two are genuine fallout of the first, but a reviewer has to hold all three at once.

Check state was the same when this review began and immediately before posting: 19 success, 1 skipped, no failures and nothing pending.

@r0x0r
r0x0r added this pull request to the merge queue Sep 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 16, 2026
@r0x0r
r0x0r added this pull request to the merge queue Sep 16, 2026
Merged via the queue into main with commit 3edfb69 Sep 16, 2026
43 of 45 checks passed
@r0x0r
r0x0r deleted the fix/dash-restore-terminal-on-sigterm branch September 16, 2026 10:53
fredespi added a commit that referenced this pull request Sep 16, 2026
…-for-the-13-readme

`3edfb693` (#326) refactored `tui_driver.rs`'s inlined final-frame drain into a
`drain_final_frame(Option<&str>)` helper, while this branch had generalized the
same loop's literal `marker` into a `matches` predicate with a `wanted` label
(so a scenario can wait on a condition a substring cannot express). Taking
either side alone would have dropped the other's work.

Resolved by keeping both: `drain_final_frame_where(wanted, matches)` carries
this branch's predicate through main's extraction, and `drain_final_frame`
stays as the substring-shaped wrapper main's second caller uses. The
post-deadline reader-panic check this branch had at the call site is dropped
because main folded the equivalent check inside the helper, and
`drain_panic_message` now takes `wanted` directly rather than re-deriving the
label from an Option.

Both predicate parameters gained `+ Sync`. The futures are held across await
points in a Send context, and a bare `impl Fn` reference is not Send unless the
closure is Sync — caught by clippy's `future_not_send` in the Linux container,
not by a local build.

Verified against a binary built from the merged tree: the mock e2e suite
reconciles 0 XPASS and 0 stale, and main's new SIGTERM/Ctrl-C dashboard
scenarios pass through the merged driver.

Signed-off-by: Frederic Espinosa <fredespi@amd.com>
sbmthakur pushed a commit to sbmthakur/rocm-cli that referenced this pull request Oct 12, 2026
…ce the host (ROCm#412)

* test(dash): hold the observation clock so gen_tps scenarios cannot race the host

dash-08/dash-09 have been red on `main` since ROCm#326 and are blocking every open
PR. They fail only on the GitHub-hosted mock lane and pass on a GPU host, which
is the tell: `tests/e2e.rs` runs 64 scenarios concurrently when no GPU is
present and exactly one when there is. ROCm#326 added five more PTY scenarios to
this feature, and the extra load on a 4-core runner pushed a pre-existing race
over the edge.

The daemon is not at fault. The CI dump shows the mock had served FOUR failed
scrapes (`failure_count: 4`) by the time the assertion read the screen: at the
2 s instance cadence that is 8 logical seconds after the last successful
observation, so clearing a value whose validity window is
clamp(3 x instance_tick, 6 s, 30 s) = 6 s was correct. The scenario simply
looked far too late and called a correct expiry a regression.

What was wrong is the assumption the scenario was built on, stated in its own
comment: that the injected logical clock "advances with daemon cycles rather
than runner wall time, so host scheduling cannot consume the validity window".
It cannot consume the *daemon's* scheduling, but the clock still advances one
gpu_tick per cycle and the cycles are paced by a wall-clock interval - so it
tracks wall time, while the harness's steps do not. Steps run on the shared
cucumber runtime, where any concurrent scenario's synchronous `run_rocm` blocks
a worker; with four workers, all of them can be blocked at once and every async
task in the process - this step's poll loop and the mock's own HTTP handler
alike - stops for seconds while the daemon child keeps ticking.

So give the hook the property the scenario always claimed. The test-only clock
file gains a `hold` directive: logical time stops at the cycle that first reads
it and moves again only when the scenario writes `hold <secs>`. dash-08/09 hold
the clock before injecting the failure, so the held observation's age is frozen
at most one instance_tick + gpu_tick (3 s) inside the 6 s window and no amount
of harness starvation can spend it; dash-09's boundary step then writes
`hold 7` to cross it deliberately. Reads that fail or do not parse keep the
directive in force, and the steps publish directives by rename, so a torn read
cannot silently resume a held clock.

With the window no longer at risk, the scenario can also stop guessing when the
TUI has caught up: instead of sleeping 50 ms and hoping, the failure step waits
for the scripted TTFT cell to leave the screen - a failed scrape clears
ttft_ms/tpot_ms - which proves the frame under assertion is one the daemon
assembled after the failure. That makes the regression test strictly stronger
than the sleep it replaces, which could pass against a daemon that had already
cleared the value if the frame had not been painted yet.

Reproduced and verified on Linux by emulating the starvation directly - an 8 s
blocking stall in the step, the daemon child ticking throughout:

  before: 2 scenarios (2 failed)   - same message and step as CI
  after:  2 scenarios (2 passed)   - and still passing with 10 s + 30 s stalls

And under the real hazard, the suite at 64-way concurrency pinned to 4 CPUs
(`-t "not @requires-gpu and not @nightly"`, same command both runs):

  before: dash-gen-tps-held-after-scrape-failure FAILED
          dash-gen-tps-expiry-boundary           FAILED
  after:  both pass; the unrelated windows/wsl/serve failures this GPU host
          cannot satisfy are unchanged

No expectations.toml row: nothing here is a known bug to be waived, and a row
would have buried a scenario that is now honest. No timeout was inflated - the
50 ms render allowance was removed, not raised.

Gates on Linux: `cargo test --workspace --all-targets`, `cargo clippy
--workspace --all-targets -- -D warnings`, `cargo fmt --all --check`,
`python3 scripts/smoke_local.py`, and the full dash feature (16 scenarios, 102
steps, all passing).

EAI-7960

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(dash): wait through the session, and read the TTFT cell by position

Review follow-ups on this PR's two new step helpers. No scenario changes
meaning; both fixes make a failing step say why it failed.

`wait_until_screen_lacks` called itself the mirror of
`TuiSession::wait_for_screen` but polled a screen it could only read, not the
session that produces it. `child` and `reader_failure` are private to
`tui_driver`, so a dashboard that died mid-wait — or a pty reader thread that
panicked — left it polling a frozen last screen for the full timeout and then
reporting "still contains marker", hiding the exit status or panic that was the
real cause.

So put the wait where the session state is: `TuiSession::wait_for_screen_where`
takes a predicate and carries `wait_for_screen`'s fail-fast arms verbatim. A
predicate rather than a second `wait_for_screen_without`, because the TTFT fix
below needs a condition that is not a substring test, and one general form beats
two near-identical loops. Both call sites move over; `gen_tps_no_longer_displayed`
keeps its substring semantics as `|s| !s.contains("tok/s")`.

`SCRIPTED_TTFT_CELL` ("50ms") was matched against the whole rendered screen, so
it was only unambiguous by luck. Any ms-suffixed value containing it as a
substring (150ms, 250ms), or a second row whose TTFT is also 50 ms, would keep
the marker present after this instance's cell had cleared — and the step would
time out blaming a scrape that had in fact landed. It is now read as an exact
cell: find the scripted model's row, count whitespace-separated fields from the
id, take `TTFT_COLUMN`. A cleared cell renders an em dash rather than nothing,
so the columns stay aligned either way.

Verified on Linux that the index is the TTFT column and not its neighbour —
pre-failure the cell reads exactly "50ms" while TPOT alongside it reads "20ms",
which is the collision a screen-wide substring could not have told apart:

    PRE-FAILURE ttft_cell=Some("50ms")
    row="│ │  TestModel/E2E-1B   10*  -  50ms  20ms  —  1/0  25%  │ │"

Gates: `cargo clippy -p e2e-cucumber --test e2e -- -D warnings`, `cargo fmt
--all --check`, dash-08/dash-09 green, and the full non-GPU suite re-run under
the hazard this PR exists to fix (64-way concurrency pinned to 4 CPUs): 113
scenarios, 0 unexpected failures.

EAI-7960

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

---------

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants