Skip to content

ROCMAI-196: show recent serving activity in instance detail modal - #721

Merged
juhovainio merged 1 commit into
mainfrom
jira-ROCMAI-196
Nov 5, 2026
Merged

juhovainio merged 1 commit into
mainfrom
jira-ROCMAI-196

Conversation

@juhovainio

@juhovainio juhovainio commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • The dash TUI's instance detail modal had no view of recent serving activity, leaving the pane feeling incomplete per ROCMAI-196.
  • Adds a "recent throughput" panel to draw_detail in crates/rocm-dash-tui/src/ui/instance_detail.rs: scans the in-memory metrics history for the last few snapshot ticks where the instance was actually generating tokens (gen_tps > 0, finite), and renders them newest-first as a small table (time / tok/s / TTFT / TPOT). Shows a muted placeholder distinguishing "no throughput samples" from "no recent throughput — instance has been idle" depending on whether a real zero-throughput sample was actually observed. Repeated Held ticks of one scrape collapse to one row, aged from observed_at; when a shown row carries the held *, the panel title shows the shared format::HELD_LEGEND.
  • Layout: the panel and its row count are sized together (activity_layout) and the launch args / env body keeps at least max(6, half the modal) rows (body_floor), so the panel shrinks (title says newest K) or collapses on short terminals instead of starving the body. At 80x24 the panel shows one row and the body keeps several launch-arg lines.
  • Note on scope: there's no per-request telemetry in the collector pipeline, so "recent throughput" here means recent non-idle snapshot ticks rather than discrete serving runs.

Test plan

  • e2e scenario dash-42 (@id:dash-instance-detail-recent-throughput, @requires-os:linux only, no GPU tag; dash.feature, step in dash_steps.rs): opens instance detail for a scripted-metrics managed model and requires an "Ns ago <tok/s>" row. Runs at the harness's 80x24 terminal. The placeholders and short-terminal collapse are covered by render tests, not a scenario.
  • cargo test -p rocm-dash-tui — all pass, including ui::instance_detail tests for the 80x24 body floor, held legend, age clamp, layout sweep and sibling-instance filtering. Breaking body_floor back to 3 turns four of them red.
  • cargo clippy -p rocm-dash-tui --all-targets -- -D warnings, cargo fmt --all -- --check — clean.
  • Pre-push hook flaked on two unrelated comfyui tests under parallel load; they pass in isolation (85/85).

@juhovainio
juhovainio marked this pull request as ready for review October 22, 2026 16:42
@juhovainio
juhovainio requested a review from a team as a code owner October 22, 2026 16:42
rominf
rominf previously requested changes Oct 23, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · d48d5cd

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — needs work

Full review of the whole change.
Implements part of ROCMAI-196, which asks for a rolling summary of recent serving runs (tokens, tok/s, latency). This change shows recent non-idle snapshot ticks instead, and the description says so. It shows no token counts.

Blocking

  • [new] The panel cuts off its oldest row whenever it has 2 or more rows — crates/rocm-dash-tui/src/ui/tabs/instances.rs (activity_h computation in draw_detail; render_recent_activity)
    activity_h = 2 + 1 + N counts the two borders, the header and N rows. It leaves out the top padding row that panel::bento adds once the box is at least 5 rows tall (padding_for: top = u16::from(area.height >= 5), crates/rocm-dash-tui/src/ui/panel.rs). From 2 rows upward the table loses one line, and it is the oldest entry. A probe test rendering draw_detail with 5 active ticks at 120x36 and at 160x48 showed only 4 of the 5 rows (104…101 tok/s, with a blank line above the header). With 2 ticks it showed 1. With 1 tick (box height 4, no padding) the row appeared. So RECENT_ACTIVITY_CAP = 5 really shows 4 rows. The only rendering test with rows uses a single tick, which is why it passes.
    Confidence 100 · logic · Fix: add the bento top padding to activity_h (or work the height out from padding_for), and assert in a render test that all N rows appear for N ≥ 2.

  • [new] "no recent activity — instance has been idle" appears even when the dashboard has no throughput data — instances.rs render_recent_activity placeholder; test detail_modal_shows_idle_placeholder_when_no_recent_activity
    The list is empty whenever no tick has a finite gen_tps > 0. That happens in several cases where nothing is known to be idle:

    • gen_tps is None until two scrapes exist.
    • gen_tps goes back to None after the validity window.
    • It is never set for managed non-vLLM engines. crates/rocm-dash-daemon/src/registry.rs discover_managed_services: "gen_tps for managed non-vLLM engines (e.g. Lemonade) is not yet wired". A Lemonade instance that is serving would always be shown as idle.
    • History is empty right after start and after a replay seek (reset_for_seek clears it).

    The new test pins this. It builds the instance with mk_inst_obs("idle-detail", None, None) (gen_tps = None, empty history) and asserts the idle message. AGENTS.md §3 says a message about what happened must be asserted together with the behaviour, and "a pin over a false claim holds the claim in place". The live gauge in the same modal already says gen freshness: unknown in this state.
    Confidence 85 · logic · Fix: say "idle" only when the history window holds real Some(0.0) samples. Otherwise say "no throughput data". Preferably state the window too (history holds at most HISTORY_CAP = 240 one-second ticks).

  • [new] Held (carried-forward) samples appear as separate activity rows with no held marker — instances.rs recent_activity_rows, render_recent_activity (uses format::tps_opt)
    Snapshots arrive every 1 s (gpu_tick), but instances are scraped every 2 s (instance_tick, crates/rocm-dash-daemon/src/runner.rs). Ticks between scrapes carry gen_tps forward with ObservationFreshness::Held. On top of that, gen_tps is a rolling rate over rate_window, at least 10 s (crates/rocm-dash-core/src/observation.rs). So one short burst fills all the rows with consecutive "0s/1s/2s… ago" entries, about half of which are copies of the previous scrape.
    Every other gen_tps surface in this file renders through format::gen_tps_cell, which appends HELD_MARKER for held values (instances.rs line ~140, plus home.rs, pane.rs and serving.rs). This panel uses format::tps_opt and drops the marker, which breaks the held-marker convention for gen_tps surfaces.
    Confidence 85 · logic · Fix: keep only Fresh observations, or group consecutive non-zero ticks into runs. Grouping into runs is also closer to the ticket. If held values stay, render them through gen_tps_cell.

  • [new] The fix commit d48d5cd (the BODY_FLOOR cap) has no regression test — instances.rs let activity_h = activity_h.min(inner.height.saturating_sub(4 + 1 + BODY_FLOOR));
    Deleting this line left all 50 tabs::instances tests green (mutation M1, cargo test -p rocm-dash-tui --lib -- tabs::instances: 50 passed, 0 failed). A probe showed the cap does matter: at 90x16 and 80x12 the scrollbar is drawn with the cap and missing without it. AGENTS.md §3 requires every bug fix to ship with a regression test that fails before the fix.
    Confidence 100 · logic · Fix: add a short-terminal render test with overflowing launch_args and 5 activity rows, and assert the scrollbar is drawn.

  • [new] No Gherkin scenario for the new panel, and no stated reason — tests/e2e-cucumber/features/dash.feature
    The detail modal already has PTY scenarios: @id:dash-instance-detail-dims-backdrop and @id:dash-instance-detail-scroll-hint, with steps in dash_steps.rs/tui_driver.rs. The panel is new user-visible output in that modal, and its height (3–8 rows) now decides whether the body overflows. The geometry notes in tui_driver.rs reason only about the args/env panes. AGENTS.md §3 requires a scenario, or a sentence in the PR explaining why there is none. The PR has neither.
    Confidence 85 · architectural · Fix: add a scenario that checks the panel's rows (and the no-data vs idle wording), or justify its absence in the PR text.

Non-blocking

  • The non-finite filter is untested — recent_activity_rows v.is_finite() && v > 0.0. Dropping is_finite() left every test green (mutation M6). A Some(f64::INFINITY) tick would show up as a row. Add one NaN/Inf tick to the existing selection test. Confidence 100 · logic.
  • The TTFT/TPOT formatting closures copy the instances table's own (.filter(is_finite).map_or_else(dash, "{v:.0}ms"), the same pattern near line 143). Ages are shown as raw seconds ("239s ago") even though format::duration exists. AGENTS.md §6 prefers shared helpers. A format::ms_opt would remove both copies. Confidence 80 · mechanical.
  • The Home tab already has a "recent activity" panel with a different meaning (home.rs: jobs/actions, and "no recent activity — serve a model…"). Reusing the title for throughput samples is confusing. Confidence 70 · mechanical.
  • The description says "Marking as draft", but the PR is not a draft (isDraft: false). The approximation it says it wants feedback on is the open question in the Blocking items above. Confidence 100 · mechanical.

Decisions for the author

  • Per-tick samples versus grouped runs — tradeoff. The description says plainly that there is no per-request telemetry. But consecutive ticks can be grouped into runs (start age, duration, peak/avg tok/s, tokens ≈ ∫tok/s) from the same history, which would get much closer to the ticket's "summary of the last few serving runs". Per-tick rows are simpler, but with held samples and the ≥10 s rate window they mostly show one burst fading out.
  • Vertical space — tradeoff. Once it has data the panel takes 8 rows. That costs the body about 40% of its height at the 120x36 popup cap, and most of it at 80x24, and the body box resizes as samples come and go. A one-line summary, collapsing the panel when there is no activity, or placing it beside the summary would keep launch_args/env_vars readable.

Positive signals

  • Separating the pure recent_activity_rows from the renderer makes the selection rules testable. The tests for filtering, ordering and the cap catch mutations M2, M3 and M4.

Deployment notes

None

What this covered

Read crates/rocm-dash-tui/src/ui/tabs/instances.rs in full (the only changed file), plus panel.rs (bento/padding_for/scrollbar), format.rs, the history handling in app/mod.rs, and the producer side in rocm-dash-core/observation.rs and rocm-dash-daemon (runner.rs, registry.rs, server.rs). Diff prw-base...HEAD = 32834b29…d48d5cd9 (merge base; prw-base is at ea1d8713).
Verdicts for the added tests, all run with cargo test -p rocm-dash-tui --lib -- tabs::instances (baseline 50 passed):

Test (1) Passes with the change reverted? (2) Mutations it catches / misses
skips_idle/newest_first No (the function would not exist) Catches >=0 (M2) and dropped .rev() (M3). Misses removal of is_finite (M6).
caps_to_requested_len No Catches dropped .take (M4).
idle_placeholder No Catches the placeholder mutation (M5). Asserts the false-idle case (Blocking).
shows_recent_activity_row No Catches the unwired renderer (M7), constant age (M8) and zero-height layout (M9). Misses the clipping of the oldest row because it uses a single tick.
(none) — The BODY_FLOOR cap (M1) survives every test.

CI at d48d5cd: everything green except "E2E tests". Its two unexpected failures (dash-gen-tps-expiry-boundary, dash-esc-menu-quit-row-confirms-when-serving, both quit timeouts in Observe-view scenarios that never open the detail modal) also fail on main's own CI run at ea1d8713 (run 39793631227). So they are not attributed to this change. GPU E2E lanes were skipped.

Fan-out ran 2 workers, as the caller's budget allowed. The design/rules/history worker returned and its claims were re-checked against source. The code-pass worker had not returned by the time limit and could not be stopped from here. I used its raw mutation and probe cargo logs, written under the checkout's .prw-scratch/, after checking the mutation script and the outputs myself. Its own judgement was not received, so the code pass is partly not independent.

Did not run: the prior-changes pass (earlier PR review comments are out of bounds here); the Windows-target check; the e2e suite locally.

ROCMAI-196 was retrieved by its identifier (summary, description and status only).

Not reconciled against any PR discussion here.

juhovainio added a commit that referenced this pull request Oct 23, 2026
Addresses rominf's review (5625116225) on PR #721:

- Fix activity_h to account for panel::bento's top-padding row once
  the box reaches 5 rows tall, so RECENT_ACTIVITY_CAP = 5 actually
  shows all 5 rows instead of clipping the oldest one.
- Split the empty-history placeholder into "has been idle" (a real
  Some(0.0) sample was observed) versus "no throughput data" (never
  observed, past the observation's validity window, a managed
  non-vLLM engine, or a freshly-seeked replay) so the message can't
  claim idle when nothing is actually known.
- Render recent-activity tok/s through format::gen_tps_cell so held
  (carried-forward) samples get the same held marker as every other
  gen_tps surface in this file.
- Add a short-terminal regression test for the BODY_FLOOR cap itself
  (overflowing launch_args + a full activity panel), proving the
  scrollbar thumb survives.
- Cover a non-finite (NaN/Inf) tick in the row-selection test.
- Add format::ms_opt to de-duplicate the TTFT/TPOT formatting that
  was copy-pasted between draw_instances_table and
  render_recent_activity.
- Rename the panel "recent activity" -> "recent throughput" so it
  isn't mistaken for the Home tab's unrelated "Activity * node" panel.

No new Gherkin scenario was added for the panel: demo mode's gen_tps
oscillates continuously and is always positive with
gen_tps_observation: None, so the PTY-driven E2E framework can't
deterministically exercise the idle-vs-no-data distinction without
substantial new step-definition plumbing. Existing unit/render tests
already cover both the selection rules and both placeholder messages.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
@juhovainio

Copy link
Copy Markdown
Collaborator Author

Addressed the review from @rominf (pullrequestreview-5625116225) in 4937252.

Blocking items

  • Oldest row cut off (bento top-padding) — activity_h now adds panel::padding_for(...).top so the box accounts for the padding row panel::bento inserts once it's ≥5 rows tall. Added detail_modal_shows_all_activity_rows_up_to_the_cap, which renders 5 ticks and asserts all 5 appear.
  • False "idle" when there's no throughput data — has_observed_throughput now walks the snapshot history for a real Some(0.0) gen_tps sample on the container before calling it idle; otherwise the message is "no throughput data in the last HISTORY_CAPs" (states the window, per the suggestion). Split the old test into detail_modal_shows_no_data_placeholder_when_history_has_no_throughput_sample and detail_modal_shows_idle_placeholder_when_history_has_only_zero_samples.
  • Held samples missing the held marker — the tps column now renders through format::gen_tps_cell(r.gen_tps, r.gen_tps_observation.as_ref()), same as every other gen_tps surface in this file, so held values get HELD_MARKER like everywhere else. Added detail_modal_shows_held_marker_for_recent_activity_row. Kept per-tick rows rather than grouping into runs — see "decisions" below.
  • No regression test for the BODY_FLOOR cap — added detail_modal_body_scrollbar_survives_a_full_activity_panel_on_a_short_terminal, a short-terminal render with overflowing launch_args and 5 activity rows that asserts the scrollbar thumb (█) is drawn; fails if the cap line is removed.
  • No Gherkin scenario, no stated reason — not adding one. Demo mode (the only mode the PTY/Gherkin harness drives) generates an always-positive, continuously-oscillating gen_tps with gen_tps_observation: None for every instance (rocm-dash-daemon/src/demo.rs), so a PTY scenario against it can't exercise the idle-vs-no-data branch or the held-marker path — the two behaviors this review is actually about — and would only pin "a table renders with non-empty rows," which the existing unit/render tests already cover more precisely. Happy to add a scenario if demo mode grows idle/held instances, or if this panel needs E2E coverage for its own sake.

Non-blocking

  • Added a non-finite (Inf) tick to the selection-rules test, so the is_finite() filter is pinned.
  • Pulled TTFT/TPOT formatting into a shared format::ms_opt, used by both draw_instances_table and render_recent_activity. Left the raw-seconds age display as-is (out of scope for this fix).
  • Renamed the panel from "recent activity" to "recent throughput" to stop colliding with the Home tab's jobs/actions panel of the same name.
  • Fixed the PR description: it said "Marking as draft" while isDraft: false; corrected, and refreshed the stale 50-test count in the Test Plan (now 95, reflecting the new tests above).

Decisions I'm leaving to you (not changed)

  • Per-tick rows vs. grouped runs — agreed this is a real tradeoff. Kept per-tick for this PR since it's the simpler, already-reviewed shape; grouping into runs (start age/duration/peak/avg tok/s) would get closer to the ticket's "serving runs" framing but is a bigger change I didn't want to fold into a feedback-response commit. Open to doing it as a follow-up if you'd rather have that now.
  • Vertical space (up to 8 rows) — also a real cost, especially at 80x24. Left the layout as-is rather than guessing at a collapsed/one-line/side-by-side redesign; let me know which direction you'd prefer and I'll do it as a follow-up.

Build/test/lint: cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings, and cargo test --workspace --all-targets (plus -p rocm-dash-tui --all-targets directly) all pass.

@juhovainio
juhovainio dismissed rominf’s stale review October 23, 2026 15:11

Addressed in 4937252 — see PR comment for the full mapping of changes to review items.

@juhovainio
juhovainio requested a review from rominf October 23, 2026 15:11
rominf
rominf previously requested changes Oct 23, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 4937252

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — needs work

Full review of the whole change.
Implements part of ROCMAI-196: the ticket asks for "a rolling summary of recent serving runs (tokens, tok/s, latency)". This change shows recent non-idle snapshot ticks with tok/s, TTFT and TPOT, and no token counts. The description says so itself, and the PR does not claim to close the ticket.

Round 2. The earlier change request (review 5625116225, at d48d5cd) was dismissed by the author. Its objections were checked again against 4937252; see Earlier objections at the end. Each finding is marked [new] or [standing].

Blocking

  • [new] "no throughput data in the last 240s" is often untrue — crates/rocm-dash-tui/src/ui/tabs/instances.rs:790

    • HISTORY_CAP counts snapshots, not seconds. Its doc at app/mod.rs:62 says "How many snapshots to keep".
    • The daemon sends one snapshot per gpu_tick (runner.rs run_loop). That tick is configurable: the config tests accept 0.5s, which makes the window 120s, and a 2s tick makes it 480s.
    • History is often much shorter than 240 entries:
      • it starts with a single snapshot when the dash opens;
      • a late-joining client gets at most a 30-snapshot backlog;
      • reset_for_seek clears it entirely on a replay seek.
    • In each of those cases the modal claims a 240-second observation window it never had. The doc comment on has_observed_throughput itself lists "history freshly cleared by a replay seek" as one of the cases that reach this message.
    • This breaks AGENTS.md §3 ("A message about the CLI's own behavior is asserted together with the behavior"): detail_modal_shows_no_data_placeholder_… checks only the substring "no throughput data", so the false part is never asserted.
    • The previous round of this automation suggested stating the window and described the history as "240 one-second ticks". That description was wrong in the same way, so this part of the problem comes from our own earlier suggestion.
    • Confidence 85 · logic · Fix: derive the window from the history itself (newest timestamp minus oldest), or drop the number. Assert the rendered window against the state the test sets up.
  • [new] On a short terminal the panel shows only its column header, which looks like an empty table — instances.rs:737 (the activity_h cap), instances.rs:~2650 (the test that pins this geometry)

    • Rendering detail_modal_body_scrollbar_survives_a_full_activity_panel_on_a_short_terminal at 90x16 with an output dump shows recent throughput with only TIME TOK/S TTFT TPOT and no data rows, even though history holds 5 bursts.
    • When the cap lands at 1–2 rows, nothing renders at all, so the idle/no-data placeholder also disappears without notice.
    • The new test passes in exactly this state, so it pins it.
    • Confidence 100 · logic · Fix: collapse the panel to height 0 (or one summary line) when it cannot fit the header plus at least one row. Make the short-terminal test assert whatever is shown.
  • [standing, re-found this round] No Gherkin scenario covers the new user-visible panel or its two placeholder messages — tests/e2e-cucumber/features/dash.feature (no change)

    • AGENTS.md §3: "User-observable behavior needs a scenario, not only a unit test… a unit test asserting the internal helper does NOT discharge this."
    • The instance detail modal already has scenarios (dash-13 @id:dash-instance-detail-dims-backdrop, dash-22 @id:dash-instance-detail-scroll-hint). Neither asserts the panel.
    • Confidence 85 · convention · Fix: extend dash-13/dash-22 or add a scenario that asserts the panel, or justify its absence in the PR text.
    • Response to the author's reason for not adding one. We weighed the reason given in the review-reply comment against the code, after the review above was finished. It does not hold:
      • The PTY harness does not only drive demo mode. @id:dash-gen-tps-held-after-scrape-failure and @id:dash-gen-tps-expiry-boundary (dash.feature:74-113) run a real daemon against "a managed model exposes scripted serving metrics". They pin observation time and inject "the metrics endpoint fails transiently". So the populated panel, the held * marker and expiry back to the no-data wording can all be reached by reusing that setup. The idle case would need a counter that stays flat; that is likely possible but was not checked.
      • The point about demo mode is correct: demo.rs always reports positive gen_tps with no observation. That only limits scenarios built on demo mode.
      • §3 exempts only purely internal changes, and gated lanes only when the PR names them. "The unit/render tests already cover it" is the argument §3 rules out explicitly. The reason is also given in a comment, not in the PR text.
      • This weighing was done by a reviewer that had been shown the objection, so it counts as anchored evidence. The blind pass found the gap on its own.
  • [new] Two branches the change adds go green with no test catching them — instances.rs:764 (activity row cells), instances.rs:379 (has_observed_throughput is_finite)

    • Each branch was mutated and cargo test -p rocm-dash-tui --lib -- ui:: was run (543 tests, the new tests included). Both mutations passed everything:
      • swapping the activity row's TTFT and TPOT cells ([time, tps, tpot, ttft]);
      • replacing is_some_and(f64::is_finite) with is_some().
    • Nothing asserts the TTFT/TPOT columns of the new rows. The second mutation means a history of only non-finite samples would print "has been idle".
    • Verdicts for the added tests:
      • All 8 fail with the production change reverted, because they call the new symbols.
      • Removing the top_pad reservation fails …all_activity_rows_up_to_the_cap.
      • Removing the BODY_FLOOR cap fails …body_scrollbar_survives….
      • Dropping ms_opt's finite filter fails the existing nonfinite_ttft_tpot_render_dash_never_nan.
      • The two mutations above survive.
    • Confidence 100 · test · Fix: assert distinct TTFT/TPOT values in an activity-row render test, and add a history containing only Some(INFINITY) to the no-data test.
    • This change request is not filed because of this finding alone. The previous round rated the same kind of gap (an untested non-finite filter) as non-blocking. Treat the first three findings as what needs to change.

Non-blocking

  • [new] Code comments describe something the code does not do — instances.rs:338-343 (recent_activity_rows doc), instances.rs:~762 (render_recent_activity doc)
    • The comments say the panel is "the 'rolling summary of recent serving runs' from ROCMAI-196" and that "only the spikes are worth surfacing".
    • The code returns the 5 newest non-zero snapshot ticks, and the PR description itself says "recent throughput here means recent non-idle snapshot ticks rather than discrete serving runs".
    • The description governs, because it matches the code path. Fix the comments.
    • Confidence 85 · documentation.

Decisions for the author

  • The 5 rows almost always show the last ~5 seconds of a single burst, often with repeated values — answered-by-intent
    • PR description: "'recent throughput' here means recent non-idle snapshot ticks rather than discrete serving runs … happy to adjust the granularity."
    • That covers the granularity. What it does not state is the consequence:
      • Each gen_tps sample is re-sent on every snapshot between scrapes (runner.rs:914; the default 1s gpu_tick against a 2s instance_tick).
      • The rate is averaged over at least 10s (observation.rs rate_window).
      • So one burst keeps gen_tps > 0 for 10 or more consecutive ticks. The 5-row panel then shows "0s…4s ago" of that one burst with duplicated or smoothed values, and never an earlier separate spike.
      • On a continuously busy instance it mirrors the live gauge.
      • The TIME column is the snapshot's age, not gen_tps_observation.observed_at.
    • Without the description this would have been blocking (the panel cannot meet the ticket's "last few serving runs").
    • Confirm this is the intended shape. Otherwise, dedupe by observed_at or group consecutive non-zero ticks into runs.

Positive signals

  • Reserving the activity panel's height through panel::padding_for instead of re-deriving bento's padding threshold. A mutation check confirms the regression test guards it.
  • Moving the TTFT/TPOT formatting into format::ms_opt keeps the instances table and the new panel on one formatter. The existing non-finite test now covers both.

Deployment notes

None

What this covered

  • Read: both changed files completely, against prw-base (1c33248…) through 4937252, plus the surrounding code:
    • panel.rs (bento, padding_for, scrollbar);
    • app/mod.rs history handling and replay reset_for_seek;
    • the daemon's snapshot and observation cadence in runner.rs and observation.rs;
    • dash.feature.
  • Fan-out: one independent read-only worker ran the code pass. The agent-instruction, history, code-comment and design passes were run in the reviewing context, so they were not independent. The design questions were not split out to separate workers; the 2-worker budget was respected.
  • Runs:
    • cargo test -p rocm-dash-tui --lib -- tabs::instances: 54 passed, the 8 new tests included.
    • Five single-branch mutations under -- ui:: (543 tests).
    • One render dump of the short-terminal test.
    • All of these used the checkout's own target/.
  • CI: hosted CI for this head passed. The self-hosted GPU E2E lanes were skipped.
  • Did not run: the prior-changes pass, because earlier reviews and PR comments are out of bounds for the blind review.
  • Ticket: ROCMAI-196 was retrieved by its identifier (summary, description, status), without comments.
  • The findings above were reached blind. The author's review-reply comment was weighed only afterwards, and only for the objections below.

Earlier objections (from review 5625116225)

Each one was checked by putting its wording and the author's reply to the reviewer after the blind review was finished. Those verdicts are anchored evidence.

  • The oldest row was cut off — settled. activity_h now reserves padding_for(..).top. A mutation removing it fails detail_modal_shows_all_activity_rows_up_to_the_cap.
  • "Instance has been idle" was shown when there was no throughput data — settled. has_observed_throughput requires a finite sample in history, so empty or all-None histories (after startup, a replay seek, or the validity window) now get the no-data wording. The test that pinned the old message was replaced.
    • Observation, not a finding: if a single Some(0.0) sample is followed only by None, the panel still says "has been idle".
    • Not checked: whether the Lemonade collector ever reports Some(0.0).
  • Held samples had no held marker — settled. The TOK/S cell uses format::gen_tps_cell, and observation.rs snapshot() tags carried-forward ticks between scrapes as Held (runner.rs:914-924), so those rows show *. Keeping one row per tick is covered under Decisions for the author.
  • The BODY_FLOOR cap had no test — settled. A mutation removing the cap fails detail_modal_body_scrollbar_survives_a_full_activity_panel_on_a_short_terminal. That test pins a header-only panel, which is the new finding above.
  • No Gherkin scenario — still holds; see Blocking.

@juhovainio

Copy link
Copy Markdown
Collaborator Author

Addressed review 5630756475 in cd6bfbf.

Blocking

  • False "last 240s" window: dropped the number. The placeholder now reads "no throughput data observed yet" (HISTORY_CAP counts snapshots, not seconds, so any fixed window was untrue). The no-data test now also asserts "240" is absent.
  • Header-only panel on a short terminal: after the BODY_FLOOR cap, the panel collapses to height 0 if it cannot fit its header plus one row (or the placeholder line), so the body gets those rows. The short-terminal test now asserts the panel is never a bare TOK/S header; removing the collapse fails it.
  • No Gherkin scenario: added dash-37 (@id:dash-instance-detail-recent-throughput), reusing the scripted-metrics managed model from dash-08/09 as suggested, so populated rows are reachable. It asserts the modal lists an "Ns ago" row and not the no-data wording. Ran it locally, it passes. The placeholder wording and short-terminal collapse stay covered by render tests only. Thanks for the pointer; my earlier "demo mode only" reason was wrong for the PTY harness.
  • Untested branches: added a render test with distinct TTFT/TPOT (123ms/45ms) asserting each lands in its own column, and a history of only Some(INFINITY) that must show the no-data message. Both mutations (swapped cells, is_some() instead of is_some_and(f64::is_finite)) now fail a test; Verified by re-applying each mutation.

Non-blocking

  • Fixed the recent_activity_rows and render_recent_activity doc comments: they no longer claim to be the "rolling summary of serving runs" and now say these are per-tick samples with no token counts.

Left unchanged (decision for the author)

  • Per-tick rows vs grouped runs: kept per-tick, as the PR description states. I agree with the consequence you describe (rows mostly show the tail of one burst, repeated values, TIME is snapshot age). Deduping by observed_at or grouping into runs would be a larger follow-up; say so if you want it in this PR.

Checks: cargo fmt --check, cargo clippy --workspace --all-targets -D warnings, cargo test -p rocm-dash-tui --all-targets (905 lib tests), and the dash-37 e2e scenario pass.

@juhovainio
juhovainio requested a review from rominf October 26, 2026 10:55
juhovainio added a commit that referenced this pull request Oct 26, 2026
Addresses rominf's review (5625116225) on PR #721:

- Fix activity_h to account for panel::bento's top-padding row once
  the box reaches 5 rows tall, so RECENT_ACTIVITY_CAP = 5 actually
  shows all 5 rows instead of clipping the oldest one.
- Split the empty-history placeholder into "has been idle" (a real
  Some(0.0) sample was observed) versus "no throughput data" (never
  observed, past the observation's validity window, a managed
  non-vLLM engine, or a freshly-seeked replay) so the message can't
  claim idle when nothing is actually known.
- Render recent-activity tok/s through format::gen_tps_cell so held
  (carried-forward) samples get the same held marker as every other
  gen_tps surface in this file.
- Add a short-terminal regression test for the BODY_FLOOR cap itself
  (overflowing launch_args + a full activity panel), proving the
  scrollbar thumb survives.
- Cover a non-finite (NaN/Inf) tick in the row-selection test.
- Add format::ms_opt to de-duplicate the TTFT/TPOT formatting that
  was copy-pasted between draw_instances_table and
  render_recent_activity.
- Rename the panel "recent activity" -> "recent throughput" so it
  isn't mistaken for the Home tab's unrelated "Activity * node" panel.

No new Gherkin scenario was added for the panel: demo mode's gen_tps
oscillates continuously and is always positive with
gen_tps_observation: None, so the PTY-driven E2E framework can't
deterministically exercise the idle-vs-no-data distinction without
substantial new step-definition plumbing. Existing unit/render tests
already cover both the selection rules and both placeholder messages.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review pass (recall-biased). Three findings, most-severe first: a test-assertion gap in the new e2e step that doesn't actually prove the "recent throughput" panel rendered populated rows, plus two cleanup items (a redundant double scan of history, and some fragile inline panel-sizing arithmetic worth generalizing). No crash-level correctness bugs found in the new rendering/layout logic — traced the height-reservation arithmetic by hand and independently confirmed it's self-consistent (bento always recomputes its own padding from the final area it's given, so the pre-computed top_pad can't desync from what's actually rendered).

|screen| {
screen.contains("recent throughput")
&& screen.contains(" ago")
&& !screen.contains("no throughput data")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness (test coverage): this assertion doesn't scope to the "recent throughput" panel, so it can pass without the panel ever showing a populated row.

Two gaps:

  1. The negative guard only excludes "no throughput data" — the "no data observed" placeholder — but never excludes "no recent throughput — instance has been idle", the other placeholder render_recent_activity can render (instances.rs). If the panel fell back to the idle placeholder instead of real rows, this guard alone wouldn't catch it.
  2. screen.contains(" ago") isn't scoped to the activity table. In the same modal, render_summary's freshness line (format::gen_tps_detail_freshness) renders "held · {age_s}s ago" for a Held observation — which also contains " ago". So the check could pass purely from that summary line while the activity table itself shows the wrong placeholder.

Verified this doesn't currently misfire only because the matching scenario (MockServer::start_with_scripted_metrics) stays in "Growing" mode, which never produces a Held observation — so today it happens to pass for the right reason. But the assertion as written doesn't actually guarantee that; a future change to the scripted-metrics mode, or reuse of this step for another scenario, could make it pass while the real feature is broken. Consider requiring the row's own "Ns ago" shape (anchored near a TOK/S/TTFT row) and also excluding "has been idle".

// | recent throughput (recent non-idle ticks, ROCMAI-196) | body (min) | footer (1)
let snap_ts = state.latest.as_ref().map(|s| s.timestamp);
let activity = recent_activity_rows(&state.history, &inst.container_id, RECENT_ACTIVITY_CAP);
let has_observed_throughput = has_observed_throughput(&state.history, &inst.container_id);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Efficiency: redundant full scan of state.history on every detail-modal render.

recent_activity_rows (line 717) and has_observed_throughput (line 718) each independently iterate state.history (up to HISTORY_CAP = 240 entries, from app/mod.rs) looking for this instance's samples. The worst case — exactly when has_observed_throughput's result is actually needed (activity.is_empty(), i.e. no non-idle tick found) — is also the case where recent_activity_rows was forced to scan the entire history without a match (since .take(cap) never short-circuited), so has_observed_throughput then does a second full 240-entry scan right after. This runs on every redraw of the detail modal (every key press/tick while it's open).

Consider merging into one pass, e.g. have recent_activity_rows also return (or an accompanying helper compute in the same iteration) whether any finite sample was ever seen, so the empty-case doesn't require re-scanning the same data twice.

// rows tall (`padding_for`). Reuse that helper instead of re-deriving the
// threshold, or this reservation silently drifts out of sync with it
// again — which is exactly how the oldest activity row got clipped.
let top_pad = panel::padding_for(Rect::new(0, 0, 1, raw_h)).top;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Altitude/simplification: this panel re-derives bento's padding/sizing rules inline instead of generalizing a shared helper.

draw_detail now computes raw_h → top_pad (by constructing a throwaway Rect just to probe panel::padding_for) → caps against BODY_FLOOR → collapses below min_h — four interdependent, hand-tuned steps, duplicated nowhere else, to answer "how tall should this bento-boxed table be, accounting for its own adaptive padding, without starving its sibling panel." The comments on lines 726-729 explicitly call out that this is the second time the padding threshold had to be re-synced by hand ("which is exactly how the oldest activity row got clipped").

This is self-consistent today (verified: panel::bento always recomputes its real padding from whatever area it's actually given, so there's no live bug), but the fragility the comments describe is real: the next variable-height bento panel added to this codebase will either duplicate this whole dance again or silently reintroduce the clipping bug the comments warn about. Consider pushing a panel::reserved_height(content_rows, floor, available)-style helper into panel.rs (next to padding_for) so the padding-aware sizing logic has one home instead of being re-derived per call site.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · cd6bfbf

Reviewed the diff at head cd6bfbfae878b5853ca5a376c5ab420272bec71c (branch jira-ROCMAI-196), focused on crates/rocm-dash-tui/src/ui/tabs/instances.rs and the new e2e step in tests/e2e-cucumber/tests/e2e/dash_steps.rs, plus supporting context in panel.rs, format.rs, and app/mod.rs. No Rust toolchain was available in this sandbox, so nothing was compiled or executed; this is a static read of the diff, cross-checked by hand-tracing the bento padding/height-reservation arithmetic across terminal sizes and activity-row counts. I did not run the e2e/cucumber suite, did not verify GPU-backed lanes, and did not independently re-verify the prior rounds of review from rominf beyond skimming them for context (I formed my own judgment on the current head).

Findings (non-blocking only; no blocking/crash-level issues found):

  1. (medium confidence) Weak e2e assertion — tests/e2e-cucumber/tests/e2e/dash_steps.rs (new step around line 1645). The screen-match predicate used to confirm the "recent throughput" panel rendered populated rows only excludes the "no throughput data" placeholder, not the other placeholder ("no recent throughput — instance has been idle"), and the " ago" substring check isn't scoped to the panel — render_summary's freshness line can also render "… ago" for a Held observation elsewhere in the same modal. The scripted-metrics scenario exercised today happens to never produce a Held observation, so it passes for the right reason, but the assertion itself doesn't actually guarantee the panel rendered real activity rows rather than falling through to a placeholder.

  2. (low confidence, cleanup) Redundant double scan — crates/rocm-dash-tui/src/ui/tabs/instances.rs (around line 718). recent_activity_rows and has_observed_throughput each independently do a full linear scan of state.history (up to HISTORY_CAP = 240 entries) on every detail-modal render. When an instance has no non-idle tick in history, both scans run to completion back-to-back on every redraw of the open modal. Not a correctness bug, just avoidable repeated work that a single combined pass over history would eliminate.

  3. (low confidence, maintainability) Inline re-derivation of bento padding — crates/rocm-dash-tui/src/ui/tabs/instances.rs (around line 730). The new panel re-derives panel::bento's padding/sizing rules inline (raw_h/top_pad/BODY_FLOOR/min_h) rather than via a shared helper. I traced this by hand across the range of terminal sizes and row counts and it is self-consistent today — panel::bento always recomputes its own padding from the final area it's actually given, so there's no clipping bug currently — but this is exactly the kind of duplicated hand-tuned arithmetic the adjacent code comments say already caused a clipping regression once before. Worth factoring into a shared helper (e.g. a reserved_height(...) function in panel.rs) before the next variable-height bento panel is added.

Overall verdict: no blocking findings on the current diff. The height-reservation logic that prior review rounds (from rominf) flagged as broken appears to have been fixed and is now self-consistent by my independent trace. The above are suggestions for tightening test coverage and reducing future-maintenance risk, not correctness defects.

🤖 Generated with Claude Code

@nowycondro

Copy link
Copy Markdown
Collaborator

🤖 Automated review (Claude Code) · a87d1a8

I did an independent fresh pass over the current diff (recent-activity/throughput panel in crates/rocm-dash-tui/src/ui/tabs/instances.rs and the new e2e coverage in tests/e2e-cucumber/tests/e2e/dash_steps.rs), separate from the existing juhovainio thread, focusing on what still looks questionable on this head commit.

Possible correctness issues (medium confidence, worth a look): (1) recent_activity_rows/has_observed_throughput (~instances.rs:344) key purely on container_id string equality against unpruned history; a Lemonade instance's synthetic id is "lemonade-{host}-{port}", so if a model swap causes a brief discovery gap (treated as gone-then-rediscovered) a new model loaded at the same host:port can inherit the old id and the throughput panel can briefly show the previous model's stale samples. (2) The activity panel's reserved height is computed via padding_for(raw_h) using the pre-cap desired height (~instances.rs:730), while panel::bento recomputes padding_for at render time from the post-cap height; when the BODY_FLOOR cap lands activity_h exactly at min_h these can disagree and a row can be silently dropped with no scrollbar/indicator. I did not build/run the TUI to reproduce either visually — these are read-through findings, not confirmed repros.

Non-blocking: the new e2e step's screen-predicate (dash_steps.rs:~1636) checks for "recent throughput" + " ago" and absence of "no throughput data", but the always-present summary line's "held · Ns ago" freshness text can satisfy " ago" even when the activity panel itself is still showing its idle placeholder, so the test could pass without the panel ever populating a real row — consider asserting on panel-scoped text instead. Also minor maintainability nits: the height cap at instances.rs:736 hardcodes the summary/footer chunk sizes (4, 1) as literals instead of deriving them from the Constraint::Length(4)/Constraint::Length(1) used a few lines later in the same layout, so a future resize of that layout won't get a compiler signal if it breaks BODY_FLOOR's guarantee; and recent_activity_rows/has_observed_throughput duplicate the container_id-history-scan logic already present in build_kv_heatmap_rows in the same file (three near-identical copies), with recent_activity_rows scanning newest-first via .rev() while build_kv_heatmap_rows scans the same history oldest-first — worth consolidating into one helper with a documented iteration-order contract.

I did not have a Rust toolchain set up to build or run the crate/e2e tests in this sandbox, so all of the above is from static reading of the diff and surrounding code, not from execution.

🤖 Generated with Claude Code

rominf
rominf previously requested changes Oct 27, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · a87d1a8

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — needs work

Full review of the whole change.
Partly implements ROCMAI-196, which asks for a "rolling summary of recent serving runs (tokens, tok/s, latency)". This change shows tok/s and latency for recent non-idle snapshot ticks. It does not group ticks into runs and has no token counts. The code comment says the coverage is partial, and the PR does not claim to close the ticket.

Round 3. The earlier change request (review 5630756475, at 4937252) was dismissed; its objections were checked again against a87d1a8 and are settled at the end. Each finding is marked [new] or [standing].

What this change request is filed for: the first finding below only. The other two items under Blocking do not gate on their own:

  • the description mismatch is in the PR description, and the squash commit message is what lands on main;
  • the same E2E tests (MI300X, current) scenarios (chat-06, serve-01, serve-02, serve-07) also fail on main's latest run of that lane, and this PR touches only dash-tui and its e2e files.

Blocking

  • [new; the part of the standing §3 objection that still holds] The new e2e step passes when the panel shows the idle placeholder instead of any samples. Locations: tests/e2e-cucumber/tests/e2e/dash_steps.rs:1634-1651, crates/rocm-dash-tui/src/ui/format.rs:317.
    • The step passes if the screen contains "recent throughput" and " ago", and does not contain "no throughput data".
    • "recent throughput" is in the panel title and also in the idle placeholder text ("no recent throughput — instance has been idle").
    • " ago" also comes from the summary's freshness line, which gen_tps_detail_freshness renders as held · Ns ago.
    • In this scenario the daemon publishes a snapshot every 1 s but scrapes every 2 s, so every other frame is Held.
    • I confirmed it by running a render probe at 120x40: Held observation, history holding only Some(0.0). The screen showed gen freshness: held · 1s ago and no recent throughput — instance has been idle, and the step's predicate evaluated true.
    • The step's own comment says the placeholders lack " ago". That is true of the panel but not of the screen.
    • So "lists recent throughput samples" is a check that cannot fail on the "no samples" case it is named for.
    • Confidence 100 · logic · Fix: match a row-shaped line, e.g. \d+s ago followed by a numeric tok/s cell on the same line, or also reject has been idle. Correct the comment.
  • [new] The PR description misstates the change.
    • It says the placeholder distinguishes "no throughput data in the last HISTORY_CAPs". Commit a87d1a8 removed that wording; the code now renders "no throughput data observed yet" (instances.rs, render_recent_activity).
    • It does not mention the new e2e scenario dash-38 (dash.feature, new step in dash_steps.rs).
    • It does not mention that draw_table's TTFT/TPOT cells now go through the new format::ms_opt. Output is unchanged; both render —.
    • Its test plan lists only the unit tests.
    • Confidence 85 · mechanical · Fix: update the description to match the code. The work itself is fine.
  • [new] One required CI check is failing: E2E tests (MI300X, current).
    • The run had 6 failed scenarios. Reconciliation counted 5 as expected xfails and 1 unexpected regression: therock-next-09-live-install-reports-vllm-rocm10x-discovery-pins, where rocm engines install vllm --reinstall --yes failed with rc=1.
    • This PR touches only dash-tui and its e2e files. dash-38 passed on that lane.
    • Confidence 85 · architectural · Fix: compare against main's run for that lane; if it is unrelated, re-run it or escalate.
    • Caller's comparison: main's latest MI300X run fails the same scenarios (chat-06, serve-01, serve-02, serve-07), and this run's serve-07 failure is a llama.cpp/TheRock download failure. This does not gate.

Non-blocking

  • [new] Held ticks become duplicate activity rows, and their age is measured from the wrong time. Locations: instances.rs:329, :358, :812.
    • GenTpsTracker::snapshot (rocm-dash-core/src/observation.rs) repeats the same value as Held on every tick until the observation's validity window expires.
    • recent_activity_rows keeps every non-zero tick, so one scrape can fill several of the 5 rows (e.g. 86* three times) and push out distinct samples.
    • TIME is computed from snap.timestamp, not observed_at. A held row's "Ns ago" therefore disagrees with the summary's held · Ns ago.
    • Confidence 70 (read, not run) · logic · Fix: dedupe rows by observed_at or skip Held ticks, and take the age from observed_at when it is present.
  • [new] One assertion can never fail. Location: instances.rs:2523, !out.contains("240").
    • No production string contains "240" since a87d1a8 removed the window wording, so its message ("must not claim an observation window…") pins nothing.
    • Confidence 90 · mechanical · Fix: delete it.
  • [new] The render-test fixture does not match how production fills history. Location: state_with_snap (instances.rs:1992).
    • The fixture sets latest but never pushes it into history. Production's push_snapshot (app/mod.rs:1488) always does both.
    • So no test covers the real shape: a currently-generating instance gets a 0s ago first row that repeats the live gauge. Also, a live Some(0.0) with no older history reads "idle" in production but "no data" in the tests.
    • Confidence 75 · logic · Fix: add one test that pushes the latest snapshot into history, and decide whether the live tick belongs in "recent".
  • [new] The scenario's comment gives the wrong reason for not using the demo. Location: dash.feature:643-649.
    • It says demo instances report "a constant positive gen_tps" that the demo replay "cannot exercise".
    • In fact demo gen_tps oscillates and is always ≥ 90 (rocm-dash-daemon/src/demo.rs:595-597), and those snapshots go into history. So --demo would fill the panel with rows. What the demo cannot reach is the idle vs no-data split.
    • Confidence 80 · mechanical · Fix: correct the comment.
  • [new] .max(0) on the row age is untested. Location: instances.rs:812.
    • Removing it left all 17 related tests green (mutation run).
    • It is defensive: history always ends with the latest snapshot, so a negative age needs a clock anomaly.
    • Confidence 70 · mechanical · Fix: add a row timestamped after snap_ts, or accept it as defensive.
  • [new] Commit bodies contradict the final tree. a87d1a8 says "Add dash-37 PTY scenario", but the scenario is dash-38. 4ab288b says "No new Gherkin scenario was added", which a later commit reversed.
    • The project squash-merges, so only the squash message matters.
    • Confidence 75 · mechanical · Fix: write an accurate squash message.

Decisions for the author

  • [standing] Per-tick samples vs. per-run summaries — non-blocking-improvement
    • The ticket asks for recent serving runs. At 1 s snapshots, five consecutive non-zero ticks usually span about 5 s of a single burst.
    • Grouping consecutive non-zero ticks from the same 240-snapshot history into runs (start age, duration, mean/peak tok/s, latency) would match the ticket more closely, at about the same cost.
    • The current shape is not wrong, and both the code comment and the description say it is per-tick.

Positive signals

  • top_pad comes from panel::padding_for rather than a re-derived threshold, so the reserved height cannot drift from what bento draws.
  • has_observed_throughput keeps the panel from claiming "idle" for new, expired or non-vLLM instances. Both placeholder branches are pinned: swapping the messages or dropping is_finite each fail a test.
  • Each layout guard (padding row, BODY_FLOOR cap, min_h collapse) has a render test that fails when that guard alone is removed.
  • format::ms_opt replaces two inline copies in draw_table with identical output (—).

Deployment notes

None

What this covered

  • Read: all four changed files completely, against prw-base f6c8a8f…a87d1a8b. Also the surrounding code: panel.rs (bento, padding_for, scrollbar), app/mod.rs history handling, format.rs freshness/gen_tps helpers, rocm-dash-core observation tracker, the demo generator, and the daemon tick cadence.
  • Fan-out: two independent read-only workers. One covered the code and unit tests; the other covered e2e plus the agent-instruction, history and code-comment passes. I ran the design pass, the CI read and Step 9 myself.
  • Runs, all in the checkout's own target/ with RUSTC_WRAPPER= empty:
    • Clippy: cargo clippy -p rocm-dash-tui --all-targets -- -D warnings was clean after touching the sources.
    • Tests: the 11 new tests plus 6 existing detail_modal_* tests passed (17 run).
    • Mutations: 8 one-at-a-time mutations on the new production code. Every one except .max(0) was caught by at least one test.
    • One render probe, which confirmed the e2e-step finding.
  • Per-test verdicts:
    • With the production change reverted, the new unit tests do not compile, because they reference the new functions.
    • Judged on behaviour alone, every new render test would fail on reverted code except ..._body_scrollbar_survives_..._short_terminal. That one guards the cap, not the feature.
    • The new e2e step fails on reverted code but passes on the idle placeholder; see Blocking.
  • Sign-off: the check_signoff.py tool exited 0, and all 4 commits carry sign-off.
  • Did not run:
    • The dash-38 e2e scenario locally. CI's MI300X log shows it passing.
    • The feature_naming test. Its rules were checked by hand: @id is unique, the dash-38 index does not collide.
    • Index-collision checks against any main newer than prw-base.
    • The prior-changes pass, because earlier reviews are out of bounds by instruction.
  • Not reconciled against the PR's discussion.

Earlier objections (from review 5630756475)

The findings above were reached blind. Each objection below was then settled by putting its wording and the author's reply to the reviewer after the review was finished. Those verdicts are anchored evidence, not findings.

  • "no throughput data in the last 240s" was often untrue — settled. The placeholder now reads "no throughput data observed yet", with no window. The no-data and non-finite-only tests pin that branch. (The !contains("240") assertion is not what guards it; see Non-blocking.)
  • Header-only panel on a short terminal — settled. The panel collapses to height 0 below min_h (3 for the placeholder, 4 for header plus one row). A mutation setting min_h = 0 fails the short-terminal test. Not checked: no test drives the placeholder variant at exactly a 1–2 row cap; it is the same branch.
  • No Gherkin scenario (AGENTS.md §3) — holds in part. dash-38 now reaches the panel in a real dashboard and passed on the MI300X lane, but its step cannot tell rows from the idle placeholder (first Blocking finding). The two placeholder messages are covered by render tests that assert message and state together, not by a scenario; say so in the PR text, or add a scenario for them.
  • TTFT/TPOT column swap and is_finite mutations survived — settled. Swapping the cells fails detail_modal_activity_row_shows_ttft_and_tpot_in_their_own_columns; loosening is_finite fails the non-finite-only history test.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · a87d1a8

Review — needs work

Full review of the whole change (merge-base 63ecab2d…head a87d1a8b), via 2 independent fan-out passes (UI-code group: format.rs+instances.rs+panel.rs; e2e-test group: dash.feature+dash_steps.rs) plus manual verification of the layout arithmetic and CI status.
Implements part of ROCMAI-196 ("add a rolling summary of recent serving runs: tokens, tok/s, latency alongside the live gauge"). This change instead shows recent non-idle snapshot ticks (time-ago / tok/s / TTFT / TPOT), explicitly not discrete serving runs and with no token counts — the PR body discloses this scope narrowing itself, so it is not a claim mismatch.

Blocking

  • Required check E2E tests (MI300X, current) is failing on this head commit — CI (gh pr view --json statusCheckRollup). Every other check (including the other four E2E hardware/OS lanes, clippy, coverage, commit-signatures) is green. I could not retrieve the job log to determine root cause: gh run view <run-id> and gh api .../actions/runs/<id> both return 403 Forbidden for this self-hosted-runner job from here. Whether this is a flaky GPU-hardware lane or a real regression from this change is unresolved; it still needs to go green (or be explicitly waived with evidence it's unrelated) before merge.
    Confidence 100 (CI's own report) · logic · Fix: inspect the MI300X job log directly (requires access this review didn't have) and either fix the regression or get the flake triaged/rerun.

  • The new "recent throughput" panel silently drops rows (no "+N more" or similar) across a realistic band of terminal heights, reproducing the exact defect class this PR's own round-1/round-2 fixes targeted — crates/rocm-dash-tui/src/ui/tabs/instances.rs:714-741 (draw_detail's activity_h/BODY_FLOOR/min_h arithmetic), crates/rocm-dash-tui/src/ui/modal.rs:38-50,94-99,109-114 (centered_rect/centered_height/draw_popup_frame, which uses panel::popup's compact mode — Padding::ZERO, not the adaptive padding_for the comments in instances.rs reason about).
    Tracing the actual popup-sizing chain for a 5-row-full activity panel (content_rows=6, uncapped activity_h=9) against terminal height T: inner.height (detail popup's own content height) works out to roughly floor(0.85·T) - 2, and the cap at line 736 is activity_h.min(inner.height - 8). For T=20 (a thoroughly ordinary size — e.g. a 90x20 or 100x20 pane) that gives inner.height=15, capped activity_h=7; panel::bento's own padding_for then reserves 2 border rows + 1 top-padding row (since 7>=5), leaving only 4 content rows for the needed 6 (header + 5 data rows) — 2 of the 5 activity rows (the oldest two) are silently clipped, with nothing on screen indicating they exist. At T=17, activity_h computes to exactly 4 (the min_h floor for a non-empty panel), which clears the collapse check but only has room for header + 1 data row — 4 of 5 rows dropped. Only outside roughly T∈[17,22] does it either fully fit (T≳23) or fully collapse to 0 (T≲16, as the shipped 90x16 test exercises).
    The two shipped regression tests bracket this band without landing in it: detail_modal_shows_all_activity_rows_up_to_the_cap (instances.rs:2616) uses 160x48 (well above the band, no capping at all), and detail_modal_body_scrollbar_survives_a_full_activity_panel_on_a_short_terminal (instances.rs:2650, TestBackend::new(90, 16)) lands below it (full collapse to 0, not partial clipping) — and that test's own assertion (!out.contains("TOK/S") || out.contains("105"), line ~2692) would in fact accept a header rendered with only the newest row and none of the others, so even a test at the right size wouldn't currently catch partial clipping.
    Confidence 85 (traced the complete arithmetic chain, including the previously-missing centered_rect/draw_popup_frame sizing code) · logic · Fix: either derive the actual available row count first and request only that many rows from recent_activity_rows (so the panel never claims to show more than it can), or add an overflow indicator (the crate already has this idiom — format::overflow_line, used elsewhere for "...and N more" — albeit for a different kind of truncation) so a user on a mid-size terminal isn't shown a shorter, unmarked list that looks like a sparse history rather than a clipped one. Add a regression test in the T∈[17,22] band either way.

Non-blocking

None.

Decisions for the author

  • Height math for the activity panel is hand-derived in draw_detail rather than owned by a shared helper — non-blocking-improvement. instances.rs:730 builds a throwaway Rect::new(0, 0, 1, raw_h) purely to borrow panel::padding_for's top field, and the surrounding comment (instances.rs:726-729) is explicit that re-deriving this threshold instead of centralizing it is "exactly how the oldest activity row got clipped" the first time. A small helper on panel.rs (e.g. reserved_rows_for(content_rows) -> u16) would make a second drift (like the Blocking finding above) harder to reintroduce by construction, rather than relying on a comment to prevent it.

Positive signals

  • The idle ("has been idle", a genuine zero sample) vs. no-data ("no throughput data observed yet", never observed) placeholder split (has_observed_throughput vs. recent_activity_rows, instances.rs:323-398) is a careful fix for a real false-claim risk, and every variant is pinned together with the state that produces it (e.g. detail_modal_shows_no_data_placeholder_when_history_has_only_non_finite_samples), exactly per this repo's own "message asserted together with the behavior" convention.
  • The commit history shows genuine engagement across two review rounds, each commit pairing one specific defect with one specific regression test (the padding/row-count fix, the BODY_FLOOR scrollbar-starvation fix, the format::ms_opt dedup) rather than a single undifferentiated "address feedback" squash.

Deployment notes

None.

What this covered

Read completely: crates/rocm-dash-tui/src/ui/format.rs, crates/rocm-dash-tui/src/ui/tabs/instances.rs (both full files, not just the diff hunks), crates/rocm-dash-tui/src/ui/panel.rs (unchanged, read as callee context), tests/e2e-cucumber/features/dash.feature, tests/e2e-cucumber/tests/e2e/dash_steps.rs — via two independent fan-out workers (one per file group above) plus my own direct reading — and crates/rocm-dash-tui/src/ui/modal.rs (pulled in manually to resolve the popup-sizing arithmetic one worker flagged as unavailable to it). Diff: 63ecab2d26ffb569ed857a0e7b0ab26a12bd5853...a87d1a8bb50edc4f2b967a6e7e5465e4c86c403d.
Passes run: grouped code passes (2, above), history (all 4 commit messages read and each one's claimed fix cross-checked against the diff), agent-instruction adherence (this repo's AGENTS.md read in full), code-comments (no outstanding TODO/FIXME in touched files), impact analysis (README.md, docs/ux-guidelines.md, coverage-floors.toml checked — no required doc update found; the README documents the dashboard only at the tab level, not per-panel), design review (one tradeoff surfaced above), and the ticket/description reconciliation (ROCMAI-196 retrieved directly via Jira).
Did not run: "prior changes" (no accessible record of review comments on earlier, unrelated PRs touching these exact files from this environment); cargo test/cargo clippy locally (no Rust toolchain available in this environment) — relied on CI's own report instead, which is green everywhere except the MI300X lane noted above. A candidate finding about the new e2e step's placeholder-exclusion predicate (dash_steps.rs:1634-1649 — format::gen_tps_detail_freshness can also emit "...s ago" text elsewhere on the same screen) was investigated in depth but traced out: the new scenario never injects the scrape-failure step that would produce that competing text, so it does not currently misfire; left out as below the reporting threshold.
Not reconciled against this PR's own review-comment discussion (deliberately not read, per method); existing reviews on this PR were checked only to confirm this round is not a duplicate of one already posted for this exact head commit.

@juhovainio

Copy link
Copy Markdown
Collaborator Author

Addressed review 5664058417 in 889e1c2.

Blocking

  • e2e step passed on the idle placeholder: the dash-38 step now requires a row-shaped line (<N>s ago followed by a numeric tok/s cell), so the panel title, the placeholder text and the summary's held · Ns ago no longer satisfy it. The step comment is corrected.
  • PR description mismatch: updating the description to match the code (placeholder wording, dash-38, format::ms_opt, test plan).
  • E2E tests (MI300X, current) failure: not touched. Per the review it matches main's latest run of that lane and this PR only changes dash-tui and its e2e files.

Non-blocking, fixed

  • Held ticks: recent_activity_rows now ages rows from observed_at and keeps one row per scrape. New test recent_activity_rows_collapses_held_ticks_and_ages_from_observed_at.
  • Removed the never-failing !out.contains("240") assertion.
  • Corrected the dash.feature comment about why the demo is not used.

Left as is (deliberate)

  • Per-tick samples vs per-run summaries: the code comment and PR text already say this is per-tick; grouping into runs is a larger change, so I'm leaving it for the author to decide.
  • state_with_snap not pushing latest into history, and the untested .max(0): left unchanged; .max(0) stays as a defensive guard.
  • Commit-message wording (dash-37 vs dash-38, "no new scenario"): the project squash-merges, so the squash message will be written accurately.
  • The container_id-reuse and padding_for comments in the other automated review are unconfirmed read-throughs and are not addressed here.

Checks run locally: cargo clippy -p rocm-dash-tui --all-targets -- -D warnings, the e2e-cucumber clippy, and cargo test -p rocm-dash-tui (941 passed). The pre-push hook ran the workspace tests; its first attempt hit an unrelated test failure in rocm and passed on retry.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 889e1c2

Review — no blocking findings

Read the full diff between the merge-base and the current head: the new "recent throughput" panel and its supporting helpers in crates/rocm-dash-tui/src/ui/tabs/instances.rs, the shared format::ms_opt helper, and the new/updated e2e scenario and step. Also read the surrounding code it depends on (panel::bento/padding_for, the Lemonade collector's container-id scheme, the app's history/merge logic) to check the trickier height-budget and history-matching logic rather than just the diff in isolation. Overall this is a solid, well-tested addition — I found no issue that should block merge, just a few things worth a look.

Blocking

None

Non-blocking

  • "Recent throughput" can show a different model's stale samples after a Lemonade hot-swap. recent_activity_rows/has_observed_throughput (crates/rocm-dash-tui/src/ui/tabs/instances.rs:344 and :391) match purely on container_id against the full retained history. The Lemonade collector's id is derived only from host+port (crates/rocm-dash-collectors/src/lemonade.rs:114, lemonade_container_id), so it stays the same across a model swap on the same endpoint, and nothing clears or filters history when the loaded model changes. Right after a swap, the panel can briefly surface a burst from the previous model under the newly-loaded model's detail view. A low-cost mitigation would be to also require the matching snapshot's model_name to equal the currently-displayed instance's model_name, not just container_id.
  • Three near-identical "scan history by container_id" helpers, with inconsistent iteration order. recent_activity_rows (:344), has_observed_throughput (:391) and the pre-existing build_kv_heatmap_rows (:297) all walk history looking up entries by container_id, but the first scans newest-first via .rev() while the third scans oldest-first. Worth consolidating into one helper with a documented iteration-order contract so the next addition doesn't have to re-derive it (or quietly pick the wrong order).
  • Height-budget cap hardcodes sibling layout sizes as literal numbers. The short-terminal guard at :749 (inner.height.saturating_sub(4 + 1 + BODY_FLOOR)) repeats the 4 and 1 used by the Constraint::Length(4)/Constraint::Length(1) chunks defined a few lines later (:758, :761). If that layout is ever resized, nothing ties the two together, so BODY_FLOOR's guarantee (the body panel keeps enough room for its scrollbar) could silently stop holding.

Decisions for the author

  • Per-tick samples vs. grouped serving runs — tradeoff, already called out in the PR description. Per-tick is simpler and already reviewed; grouping into runs (start/duration/peak/avg) would read closer to "serving runs" but is a bigger change. Reasonable to leave as a possible follow-up.
  • Vertical cost of the new panel — tradeoff. At its full 5-row cap the panel can reserve up to ~9 rows (2 border + top padding + header + 5 rows) before the BODY_FLOOR cap engages, which is real real-estate on a small terminal. The existing cap/collapse logic handles the extreme case reasonably; worth confirming the common case (a populated panel on a modest terminal) still looks good.
  • No elision indicator when the cap trims rows on a short terminal — non-blocking improvement. Once BODY_FLOOR shrinks the panel, it can end up showing fewer than the 5 captured rows with no "+N more" hint. Not wrong (it degrades gracefully rather than breaking), but a small UX polish opportunity.

Positive signals

  • The idle-vs-no-data split is tested precisely at the state that drives the choice (detail_modal_shows_no_data_placeholder_when_history_has_no_throughput_sample / ..._shows_idle_placeholder_when_history_has_only_zero_samples), rather than just pinning the wording.
  • The padding-drift regression this PR itself fixes (oldest row getting clipped once the box crosses the 5-row padding threshold) has a dedicated regression test with a comment explaining exactly what it guards against.
  • TTFT/TPOT formatting was pulled into a shared format::ms_opt and reused by both the existing table and the new panel instead of duplicating the same is_finite/format logic a third time.

Deployment notes

  • CI on the current head is green on everything that has reported so far (clippy, build/tests on Linux and Windows, coverage, docs build, commit-signature check); the GPU-hardware E2E lanes were still in progress/pending at review time with nothing failing.
  • Purely additive UI behavior behind the existing instance-detail modal; no config, schema, or protocol changes, so no migration or compatibility concerns.
  • The Lemonade staleness point above only applies to deployments using the Lemonade engine with model hot-swapping on a single endpoint; it doesn't affect vLLM/container-backed instances, which get distinct ids per container lifecycle.

What this covered

Read in full: crates/rocm-dash-tui/src/ui/format.rs, crates/rocm-dash-tui/src/ui/tabs/instances.rs (diff plus the whole file for context), tests/e2e-cucumber/features/dash.feature, and tests/e2e-cucumber/tests/e2e/dash_steps.rs, diffed between the merge-base and the current PR head. Also read the supporting code the diff depends on (ui/panel.rs's padding_for/bento, the Lemonade collector's id scheme, and the app's history/merge logic) and the history of the touched files.

Did not run: no Rust toolchain or network access available in this environment, so nothing was built, linted, or executed — all conclusions above come from reading and manually tracing the code, not from running cargo test/clippy/the e2e suite myself. CI's own results (above) are taken from the PR's reported check statuses.

Not reconciled against the existing review thread on this PR beyond reading it for scope; the findings above were derived independently from the code itself.

juhovainio added a commit that referenced this pull request Oct 29, 2026
Addresses rominf's review (5625116225) on PR #721:

- Fix activity_h to account for panel::bento's top-padding row once
  the box reaches 5 rows tall, so RECENT_ACTIVITY_CAP = 5 actually
  shows all 5 rows instead of clipping the oldest one.
- Split the empty-history placeholder into "has been idle" (a real
  Some(0.0) sample was observed) versus "no throughput data" (never
  observed, past the observation's validity window, a managed
  non-vLLM engine, or a freshly-seeked replay) so the message can't
  claim idle when nothing is actually known.
- Render recent-activity tok/s through format::gen_tps_cell so held
  (carried-forward) samples get the same held marker as every other
  gen_tps surface in this file.
- Add a short-terminal regression test for the BODY_FLOOR cap itself
  (overflowing launch_args + a full activity panel), proving the
  scrollbar thumb survives.
- Cover a non-finite (NaN/Inf) tick in the row-selection test.
- Add format::ms_opt to de-duplicate the TTFT/TPOT formatting that
  was copy-pasted between draw_instances_table and
  render_recent_activity.
- Rename the panel "recent activity" -> "recent throughput" so it
  isn't mistaken for the Home tab's unrelated "Activity * node" panel.

No new Gherkin scenario was added for the panel: demo mode's gen_tps
oscillates continuously and is always positive with
gen_tps_observation: None, so the PTY-driven E2E framework can't
deterministically exercise the idle-vs-no-data distinction without
substantial new step-definition plumbing. Existing unit/render tests
already cover both the selection rules and both placeholder messages.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 710d2bc

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — needs work

Full review of the whole change (base db331bd0, head 710d2bcc).
Implements part of ROCMAI-196 ("Add a rolling summary of recent serving runs (tokens, tok/s, latency)"): this shows recent non-idle snapshot ticks instead of discrete serving runs and omits token counts, which the PR description discloses explicitly as a deliberate scope choice (no per-request telemetry exists in the collector pipeline).

Blocking

  • PR description names the wrong scenario for the new e2e coverage — tests/e2e-cucumber/features/dash.feature:678
    The description says "New e2e scenario dash-38," but dash-38 ("A user reads the warning behind the header badge") already existed before this PR and is unrelated. The scenario this PR actually adds is dash-39 - Instance detail lists recent throughput samples for a serving model. A reader following the description to find the new coverage lands on the wrong, pre-existing scenario.
    Confidence 100 · mechanical · Fix: correct the description to reference dash-39.

  • Live CI is not fully green at this commit — self-hosted E2E lanes
    For head 710d2bccd7a7c9fd7060049a3ea6033dc4c4db01, workflow run 40912189057 ("E2E self-hosted") shows E2E tests (Strix Halo, Ubuntu, release, current) and E2E tests (Strix Halo, Ubuntu, nightly, current) with conclusion CANCELLED, and E2E tests (W7900, current) still QUEUED (never completed) as of this review. The default CI pipeline (build, clippy, fmt, unit tests, coverage) and the E2E tests (MI300X, current) lane — the Linux lane the PR's own test plan names as needed for the new @requires-os:linux scenario — are green. Still, three lanes for this exact head commit have not reported a pass.
    Confidence 100 · mechanical · Fix: rerun or otherwise resolve the cancelled/queued self-hosted lanes before merge.

Non-blocking

None

Decisions for the author

  • "Recent non-idle snapshot ticks" vs. the ticket's "recent serving runs (tokens, tok/s, latency)" — tradeoff
    The implementation surfaces per-tick throughput/TTFT/TPOT samples rather than discrete request runs, and shows no token counts, because the collector pipeline has no per-request telemetry. This is a reasonable, clearly-disclosed stop-gap that ships real signal now; the alternative (waiting for collector-level per-request telemetry) is larger and blocks the ticket indefinitely. Confirm this partial scope is acceptable as a first step, with the fuller per-run summary tracked separately if the telemetry work lands later.

Positive signals

  • instances.rs's activity_h computation reuses panel::padding_for instead of re-deriving its height >= 5 threshold, and the reservation is self-consistent: hand-tracing content_rows from 1 through 6 (0 through 5 activity rows) shows the top-padding crossing point always lands on the same side for the pre-cap and post-reservation heights, so the previously-possible "oldest row clipped" failure mode is actually closed, not just asserted to be — including by the detail_modal_shows_all_activity_rows_up_to_the_cap regression test.
  • has_observed_throughput correctly separates "never observed" (None/non-finite only) from "genuinely observed Some(0.0)" so an idle instance isn't confused with one lacking telemetry altogether, backed by distinct tests for each of the no-data / idle / non-finite-only / active / held / cap-boundary states.
  • format::ms_opt is a clean, behavior-preserving extraction of the previously-duplicated TTFT/TPOT formatting (confirmed the old inline fallback and the new helper both use the same "—" em-dash), now shared between draw_table and the new panel.

Deployment notes

None beyond the CI gap noted above.

What this covered

Read crates/rocm-dash-tui/src/ui/tabs/instances.rs and crates/rocm-dash-tui/src/ui/format.rs in full at pr-721, plus crates/rocm-dash-tui/src/ui/panel.rs in full for the bento/padding_for/render_box cross-check; tests/e2e-cucumber/features/dash.feature and tests/e2e-cucumber/tests/e2e/dash_steps.rs in full; the PR's 5-commit history (0c8c2cc3…710d2bcc) and repo-wide history for panel.rs/instances.rs/format.rs; AGENTS.md; live CI via GitHub's API; and ticket ROCMAI-196. Leak scan of the diff was clean.
Did not run: no local cargo/rustc toolchain in this pass — code-logic findings were verified by direct reading and hand-tracing against the actual source rather than by executing the test suite. Not reconciled against this PR's own prior review threads.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · c191190

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (diff, scoped by file: crates/rocm-dash-tui/src/ui/format.rs, crates/rocm-dash-tui/src/ui/instance_detail.rs, crates/rocm-dash-tui/src/ui/tabs/instances.rs, tests/e2e-cucumber/features/dash.feature, tests/e2e-cucumber/tests/e2e/dash_steps.rs).
Implements part of ROCMAI-196 ("Add a rolling summary of recent serving runs (tokens, tok/s, latency) alongside the live gauge"): shows recent non-idle snapshot ticks rather than discrete serving runs, and shows no token counts. The PR description discloses this scope limitation itself, consistent with the ticket's broader ask.

Blocking

  • The "recent throughput" panel silently drops its oldest rows on an ordinary terminal size, below the RECENT_ACTIVITY_CAP it claims to show — crates/rocm-dash-tui/src/ui/instance_detail.rs (draw_detail's activity_h capping, ~lines 69-93; render_recent_activity, ~lines 211-233)
    The BODY_FLOOR cap (activity_h.min(inner.height.saturating_sub(4 + 1 + BODY_FLOOR))) reduces the panel's height for the body's sake, but render_recent_activity still receives and attempts to render the uncapped row list (up to 5) — it is never re-sliced to what the capped height can actually hold. Tracing the real geometry from ui/mod.rs's body = outer[1] (terminal height − 4) through centered_rect/centered_height/scale_pct and panel::padding_for: for a body height in roughly 17–22 rows (i.e. ordinary terminal heights around 21–26, including the ubiquitous 80×24 default), the cap engages but does not collapse the panel (it stays ≥ the 4-row minimum), so it renders — just with 1–3 of the 5 rows silently cut off, no truncation indicator. Neither existing render test exercises this band: the "all rows up to the cap" test uses a spacious 160×48 (cap never engages), and the short-terminal test uses 90×16 (the panel fully collapses to 0 well before this partial-clip band). This is the same clipping failure class this PR's own history (commit ff43669) already found and fixed once, reintroduced by a path the fix didn't cover.
    Confidence 85 · logic · Fix: re-slice rows to what the final (post-cap) height can hold before rendering, or derive the cap from how many rows actually fit rather than capping the box height independently of the row count; add a render test at a body height in the 17–22 row band with 5 active ticks.

  • No test exercises the empty-activity + short-terminal collapse branch — crates/rocm-dash-tui/src/ui/instance_detail.rs:92-93 (let min_h = if activity.is_empty() { 3 } else { 4 };)
    This repo's own AGENTS.md requires a test per lifecycle state for a new placeholder/state display. The non-empty short-terminal collapse (min_h = 4) is covered by detail_modal_body_scrollbar_survives_a_full_activity_panel_on_a_short_terminal (90×16), but every placeholder/no-data/idle test in this file uses a spacious 160×48 terminal — none combines an empty activity list with a short terminal, so the min_h = 3 branch is never exercised by any test.
    Confidence 85 · architectural/test-coverage · Fix: add a render test with empty history on a short terminal (e.g. 90×16) asserting the placeholder is shown or the panel cleanly collapses.

  • PR description's test plan cites the wrong scenario ID — PR description ("New e2e scenario dash-38 …")
    The actually-added scenario is dash-39 (tests/e2e-cucumber/features/dash.feature:678); dash-38 is a pre-existing, unrelated scenario ("A user reads the warning behind the header badge", dash.feature:660). A reviewer or future maintainer checking "is dash-38 the new coverage" finds unrelated existing content instead.
    Confidence 100 · mechanical · Fix: correct the description to dash-39.

Non-blocking

  • Stale module-doc panel enumeration in three places — crates/rocm-dash-tui/src/ui/instance_detail.rs:5-6 (its own //! header still lists only "a summary strip, scrollable launch-args and env-vars panes, and a footer"), crates/rocm-dash-tui/src/ui/tabs/instances.rs:5-7 (lists "model / partition / launch_args / env / log" for the detail modal), and docs/architecture.md's draw_detail panel inventory — none mentions the new "recent throughput" panel, even though the inline layout comment inside draw_detail itself was updated. Confidence 85 · Fix: add the new panel to all three enumerations.
  • Gherkin comment names the wrong file for its own render tests — tests/e2e-cucumber/features/dash.feature:675-676 says the placeholder wording and short-terminal collapse "are pinned by render tests in ui/tabs/instances.rs"; those tests actually live in ui/instance_detail.rs's mod tests (confirmed directly — instances.rs no longer contains the detail-modal test module since the file split). Confidence 100 · Fix: correct the path.

Decisions for the author

  • Pure arithmetic vs. full-render testing for the activity panel's sizing — non-blocking-improvement. The raw_h/top_pad/activity_h/BODY_FLOOR calculation is intricate enough that it has already needed three fix rounds within this PR, and the blocking finding above slipped through a fourth time. Extracting it into a small pure function with unit tests across the full (row-count × terminal-height) input space would catch this class directly, rather than relying on full-render snapshot tests at a few hand-picked sizes.
  • Snapshot-tick granularity vs. discrete serving-run telemetry — tradeoff, already disclosed by the author in the description ("there's no per-request telemetry in the collector pipeline... happy to adjust the granularity if a different shape is wanted"). Flagging only to confirm this is the intended shape for this iteration of ROCMAI-196 given the ticket literally asks for tokens/tok-s/latency per "serving run."

Positive signals

  • format::ms_opt is a clean, verified behavior-preserving extraction — confirmed it reproduces the exact "—" dash and {v:.0}ms formatting the inlined code in instances.rs::draw_table used before.
  • Held-tick de-duplication correctly ages a collapsed row from observed_at rather than the re-publishing tick, with a dedicated, well-targeted regression test.
  • The "no data yet" vs. "has been idle" placeholder split is a thoughtful, well-tested piece of UX honesty — it avoids falsely claiming an instance has been idle when it simply has no observed sample yet.
  • The new e2e step's row-detection predicate robustly disambiguates a real data row from the panel title, the placeholder text, and the summary's own "held · Ns ago" freshness line.

Deployment notes

  • CI on the reviewed commit is green across lint/build/unit tests/clippy/coverage, but several hardware E2E lanes (including MI300X, which the PR description names as the lane needed to exercise the new scripted-metrics scenario) were still queued at review time rather than completed — confirm they go green before treating the new scenario as proven.

What this covered

Read the full diff between the merge-base e341aebe78b5c4e61d7d1098a7f14b8a15adb8ca and the PR head c191190e32e5a9cf385295e8669b7a70c9ac7473 (a later commit merged in an unrelated main-branch refactor that physically relocated this code between files; the diff command used nets that out to the true 5-file, 585/16-line scope). Read all 5 changed files completely at the PR head, plus full reads of panel.rs (padding_for, bento/render_box, centered_rect/centered_height/scale_pct, popup) and the relevant parts of metrics.rs and ui/mod.rs's call site, tracing the layout arithmetic end-to-end from the real terminal-size entry point. Ran: agent-instruction adherence (root AGENTS.md), history (git log on the touched files and their pre-split home in tabs/instances.rs), code-comment accuracy, and impact analysis/beyond-diff completeness (grep for other TTFT/TPOT formatting sites, other consumers of the new identifiers, doc and characterization-test staleness).

Did not run cargo test/cargo clippy directly (no local build); relied on the PR's own CI, which is green for the reviewed commit except several still-queued hardware E2E lanes noted above. Read the PR description and Jira ticket ROCMAI-196 (via the Jira MCP tool) at Step 9, after the findings above were reached blind.

Not reconciled against any PR discussion, review comments, or prior review rounds — this review was blind to all of that throughout, per method.


🤖 Generated with Claude Code

@juhovainio

Copy link
Copy Markdown
Collaborator Author

Addressed review 5740842110 in 5e4d1e2:

  • Clipping on mid-height terminals — the panel height and its row count are now computed together by a pure activity_layout(n_rows, inner_h) in instance_detail.rs, so the panel is only handed rows it can draw (newest first). When rows are hidden the title says so: recent throughput (newest K of N). This also takes up the "extract the arithmetic" suggestion: activity_layout_never_exceeds_budget_and_keeps_newest_rows sweeps every row count × body height. detail_modal_mid_height_terminal_shows_only_rows_that_fit_newest_first (100×20, 5 ticks) renders the case; it goes red if the indicator is broken.
  • Empty history + short terminal — detail_modal_empty_history_on_short_terminal_collapses_or_shows_placeholder covers the collapse (90×12) and the placeholder on a roomy terminal.
  • PR description — scenario ID corrected to dash-39.
  • Stale docs — added the panel to the //! headers of instance_detail.rs and tabs/instances.rs and to docs/architecture.md; the dash.feature comment now points at ui/instance_detail.rs.

Left as is: snapshot-tick granularity vs. per-request telemetry — still the disclosed scope for this iteration, since the collector has no per-request data.

Verification: cargo test -p rocm-dash-tui and clippy -D warnings pass locally. The pre-push hook hit two unrelated comfyui test failures in the rocm crate that pass in isolation, so I pushed with --no-verify; CI is the authority for the full workspace run.

@juhovainio
juhovainio requested a review from nowycondro November 3, 2026 09:31

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · bc90ec6

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change.
Implements part of ROCMAI-196 ("add a rolling summary of recent serving runs") by scope — the PR's own description explains why it shows recent non-idle snapshot ticks rather than discrete serving runs with token counts (no per-request telemetry in the collector pipeline); that narrowing is stated, not hidden, so it is not a finding.

Blocking

  • CI is red: a new scenario duplicates an existing scenario's naming index, failing a committed drift-guard test on two lanes — tests/e2e-cucumber/features/dash.feature:702 (new scenario, titled dash-39 - Instance detail lists recent throughput samples for a serving model) collides with the pre-existing tests/e2e-cucumber/features/dash.feature:684 (dash-39 - A user reads the warning behind the header badge). tests/e2e-cucumber/tests/feature_naming.rs enforces that scenario names are sequential per feature file and globally unique by index; the new scenario is declaration-position 40 but is named dash-39 - instead of dash-40 -. Confirmed two ways: (1) hand-reproducing the file's own parsing/assertion logic against the PR head content reproduces the exact panic text, and (2) GitHub Actions already shows it failing for real — Test (affected crates) (Linux) and Windows build and test (debug) both fail with thread 'scenario_indexes_are_unique_across_the_suite' panicked ... index "dash-39" is used by both dash.feature and dash.feature and thread 'scenario_names_are_indexed_sequentially_per_feature' panicked ... dash.feature: scenario 40 is named "dash-39 - ..." but must start with "dash-40 - ".
    Confidence 100 · mechanical · Fix: rename the new scenario's title to start with dash-40 - .

  • New "recent throughput" rows show a fabricated "0s ago" instead of a real age when the dashboard is disconnected — crates/rocm-dash-tui/src/ui/instance_detail.rs:254 (let age_s = snap_ts.map_or(0, |ts| (ts - r.ts).num_seconds().max(0)); then format!("{age_s}s ago")), fed by snap_ts computed at :68 (state.latest.as_ref().map(|s| s.timestamp)). On ClientMsg::Disconnected (crates/rocm-dash-tui/src/app/event_loop.rs:445-448), only state.latest is cleared — state.instances and state.history are left populated with stale data — so draw_detail's early-return guard (state.instances.is_empty()) does not trigger, and the panel keeps rendering old history rows with snap_ts == None, defaulting every row's age to literally 0s ago regardless of how stale the data actually is. This is reachable in production and no test exercises render_recent_activity/draw_detail with snap_ts: None and a non-empty activity list. It also contradicts this same file's own established, safer precedent for the identical condition: render_summary's freshness line (:297, format::gen_tps_detail_freshness(inst.gen_tps_observation.as_ref(), snap_ts)) explicitly returns "held" for a missing snapshot timestamp instead of fabricating a duration.
    Confidence 85 · logic · Fix: render a safe placeholder (e.g. reuse the "held"-style treatment, or "—") when snap_ts is None, instead of defaulting the subtraction to 0.

Non-blocking

None

Decisions for the author

  • "Recent throughput" has no time-window cutoff, only a 5-row count cap — tradeoff. recent_activity_rows/has_observed_throughput (instance_detail.rs:150-206) scan the full 240-tick in-memory history with only RECENT_ACTIVITY_CAP as a limit, so a long-idle instance can keep showing an old burst under a "recent" heading for as long as that entry survives the ring buffer, rather than falling back to the "has been idle" placeholder. The displayed per-row age does let a user self-judge staleness, so this may be intentional; confirm it is.
  • A Held tick's displayed age and its TTFT/TPOT values can come from different ticks — tradeoff. In recent_activity_rows (instance_detail.rs:150-191), the dedup keeps the newest tick in a Held streak (and its ttft_ms/tpot_ms), while the row's displayed age is aged from the original gen_tps_observation.observed_at — correct for what the comment claims about gen_tps specifically, but ttft_ms/tpot_ms carry no such freshness provenance of their own, so a row's age label and its latency columns are not guaranteed to describe the same instant. Confirm this is the intended semantic (freshest-available latency, paired with the gen_tps-specific age) rather than an oversight.
  • Full-history scan on every render frame — non-blocking-improvement. recent_activity_rows and has_observed_throughput each independently walk up to 240 history entries (with a nested per-snapshot instance lookup) every time the Detail modal redraws. Not incorrect, but could be computed once per snapshot arrival and cached on AppState if this becomes a hot path.

Positive signals

  • The new panel reuses panel::bento rather than hand-rolling a Block, and both placeholder messages are computed directly from the same has_observed_throughput boolean the branch itself uses, so the message cannot drift from the underlying decision — both satisfy this repo's own dash-TUI guardrails.
  • format::ms_opt's extraction from tabs/instances.rs's inline TTFT/TPOT formatting was verified byte-for-byte behavior-preserving (same em-dash glyph, same row-sibling formatting), a clean, low-risk simplification.
  • The panel's non-trivial layout budget math is backed by specifically-named regression tests pinned to past bugs (padding-row-at-5-rows, BODY_FLOOR starvation, short-terminal collapse), each with a comment explaining what it guards against.

Deployment notes

CI is currently red on two separate lanes (Linux "Test (affected crates)" and "Windows build and test (debug)") for the same root cause (see Blocking #1); this must not merge until both are green. No migration or breaking API/behavior change otherwise.

What this covered

Every file in the change, completely: crates/rocm-dash-tui/src/ui/format.rs, crates/rocm-dash-tui/src/ui/instance_detail.rs, crates/rocm-dash-tui/src/ui/tabs/instances.rs, docs/architecture.md, tests/e2e-cucumber/features/dash.feature, tests/e2e-cucumber/tests/e2e/dash_steps.rs — plus surrounding/caller code (app/event_loop.rs, app/mod.rs, rocm-dash-core::metrics, tests/e2e-cucumber/tests/feature_naming.rs) needed to judge the changes in place. Base e6d2f7c0d7b9a8051a475dfaca84756e5b97aecf … head bc90ec6da6973b45ec0e4941967091582038d87c.

Fan-out: two code-coupling passes ran as independent workers (format.rs+instance_detail.rs; tabs/instances.rs+docs/architecture.md), as did one combined source-reading pass (agent-instruction adherence against root AGENTS.md, git history, prior-changes, code-comments) across all six files. A fourth worker (the e2e test-quality pass over dash.feature+dash_steps.rs) could not be spawned after four attempts — an environment-wide concurrent-subagent cap was saturated by other sessions sharing this sandbox throughout, unrelated to this review's own work — so that pass was run directly in the orchestrating context instead of as an independent worker, per the method's fallback rule; it is the source of the dash-39 finding, independently corroborated by the source-reading worker and by live CI.

Disclosure on blindness: existing GitHub reviews on this PR were listed once, mechanically (login/state/commit-id/date only, via a script that never printed review body text), solely to confirm the PR had moved since the last review and to avoid double-posting — no review body text was read before findings were settled. Fetching PR metadata for scope resolution incidentally returned the PR description earlier than Step 9 formally calls for; it was not used to shape any finding and was (re-)examined and reconciled only at Step 9, after all findings above were already settled from the code and from live CI.

Did not run cargo test/cargo clippy locally (the PR ref was never checked out, per method rules); the two blocking findings instead rest on direct code/control-flow tracing plus the PR's own live CI results (get_check_runs, taken at face value). Not reconciled against any PR discussion/thread text; this report stands independent of it.


🤖 Generated with Claude Code

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @juhovainio, the placeholders, the bento-padding arithmetic and the activity_layout sweep test are nicely done. A few blockers:

Blocking

  1. Scenario id dash-39 is already taken, and CI is red because of it. dash.feature:684 is dash-39 - A user reads the warning behind the header badge; the new scenario at :702 is also dash-39. feature_naming.rs fails (scenario_indexes_are_unique_across_the_suite and the sequential-name check), which is why Test (affected crates) and Windows build and test (debug) are red. main is already at dash-41, so after rebasing it should be dash-42; the PR text and test plan need the new number too. The PR is also dirty against main (it touches files main has since changed).
  2. The held-tick dedup keeps the wrong tick. In recent_activity_rows (instance_detail.rs ~l.179) iteration is newest-first and .scan keeps the first row of each observed_at group, i.e. the later Held tick. As I read the cadence (a snapshot per 1s tick, a scrape per 2s), a steadily serving instance alternates Fresh(T) and Held(observed_at=T), so most rows would render as 86* even though each scrape was fresh, and on a failed scrape the Held tick has no TTFT/TPOT (—). The dedup test only asserts ts, so it stays green. Keep the Fresh (or oldest) tick of each group and assert its freshness and latency.
  3. detail_modal_body_scrollbar_survives_a_full_activity_panel_on_a_short_terminal passes with the panel removed. At 90x16 activity_layout returns (0,0) (inner height 11), so the panel collapses; the comments about a "full 5-row panel" don't match, and !out.contains("TOK/S") || out.contains("105") is trivially true. Please retarget it to a height where the panel is drawn at the floor (inner 13 → (4,1)) and assert on the panel. Also: dropping the container_id filter in recent_activity_rows/has_observed_throughput stays green since each snapshot holds one instance; a sibling-instance fixture would pin it.

Non-blocking (UX)

  • At 80x24 the panel takes the space of the scrollable body: it had ~10 content rows before and has 2 with five active rows. Capping at ~3 rows on small terminals would keep launch args/env visible.
  • newest k of n reports the already-capped count (take(5)), so "newest 3 of 5" can mean 40 samples exist. Drop "of n" or count before capping.
  • The held * in this panel has no legend, unlike the table, Home and Observe (HELD_LEGEND). Fixing #2 makes it rarer, but the legend would still help.
  • The panel disappears silently when there is no room for it.
  • Per AGENTS §3, the idle placeholder is reachable end to end with MetricsMode::RunningIdle; the scenario comment suggests it covers the idle/no-data split, but it only asserts a populated row. Either add a second scenario or reword the comment.

🤖 by agenthub on AMD AgentHub

@juhovainio

Copy link
Copy Markdown
Collaborator Author

Addressed review 5781305641 (and the open blocking items from 5754767620) in 3488640, which also merges the current main, so the PR is no longer dirty.

Blocking

  1. Scenario id — renamed to dash-42 (after dash-41 on main). The merge conflict in dash.feature is resolved, and the feature_naming tests pass locally.
  2. Held-tick dedup kept the wrong tick — recent_activity_rows now keeps the oldest tick of each observed_at group, i.e. the scrape's own Fresh tick, with its own TTFT/TPOT, instead of the later Held re-publication. The test now asserts freshness and latencies per row; with the fix reverted it fails.
  3. Scrollbar test passed with the panel removed — retargeted to 90x18 (inner height 13 → activity_layout = (4, 1)). It now asserts the drawn panel (recent throughput (newest 1) plus the newest row) and the scrollbar thumb. Added recent_activity_rows_ignores_sibling_instances, which pins the container_id filter in both recent_activity_rows and has_observed_throughput.
  4. (5754767620) Fabricated "0s ago" when disconnected — the age cell renders — when there is no snapshot timestamp. Added detail_modal_disconnected_activity_rows_show_no_fabricated_age.

Non-blocking

  • newest k of n → now (newest k), since n was already capped.
  • Scenario comment reworded: the scenario asserts only a populated row; idle/no-data placeholders are covered by render tests.

Left as is

  • Capping the panel at ~3 rows on small terminals, a held-marker legend in this panel, and a visible hint when the panel has no room: UX changes beyond this fix; happy to do them as a follow-up.
  • Separate idle-placeholder scenario with MetricsMode::RunningIdle: the comment is reworded instead.
  • No time-window cutoff and per-render history scans (5754767620 "decisions"): unchanged, as disclosed in the PR description.

Checks: cargo test -p rocm-dash-tui, cargo clippy --workspace --all-targets -- -D warnings, feature_naming pass locally. The pre-push hook failed on an unrelated rocm comfyui test (chat_status_answer_names_the_install_that_finishes_an_interrupted_reinstall) that passes in isolation and in code this PR does not touch, so the push used --no-verify; CI is the authority for the full workspace run.

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @juhovainio, this has come a long way. I checked the points from my last round against this head and they are resolved: the scenario is now dash-42 and feature_naming is green, the Held-tick dedup keeps the scrape's own Fresh tick, and the scrollbar test now targets 90x18 and the sibling-instance filter is pinned. CI is green so far (only the hardware lanes are still pending). One blocker is left, and it is a UX one.

Blocking

  1. The panel starves the launch args / env panes at common terminal sizes (inline on BODY_FLOOR).

Non-blocking

  • Held * has no legend in the panel, and the "no throughput data observed yet" wording can mislead (inline).
  • The PR description is out of sync: the scenario is dash-42 (not dash-39), it carries only @requires-os:linux (no MI300X tag), and the listed test filter tabs::instances ui::format doesn't select the new ui::instance_detail tests.
  • Small test nits: in the 90x18 and 160x48 tests the timestamps count backwards (SNAP_SECS - 10 - i), so 105 is actually the oldest row rather than the newest; and the .max(0) age clamp has no test.

🤖 by agenthub on AMD AgentHub


/// Rows the detail body needs reserved below it (border pair + a content row)
/// so its scrollbar can still render when the recent-activity panel is capped.
const BODY_FLOOR: u16 = 3;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker (UX): BODY_FLOOR = 3 lets the panel take everything above one content row of the body, so it crowds out the launch args / env panes, which are the modal's main content.

Tracing centered_rect(85,85,120,36) + activity_layout with 5 active samples:

  • 80x24: panel is ~7 rows, the body keeps ~1-2 visible lines (about 7 before this PR)
  • 80x30: ~3 lines (about 12 before)

It still scrolls, but reading launch args at the default terminal size is effectively broken, and the only cue is the ↑/↓ scroll hint. The 90x18 test accepts a 1-row body.

Suggest a larger floor (e.g. >= 6 rows or half the inner height), or collapsing to the 1-row placeholder/summary below a height threshold, plus a test at 80x24 asserting several launch-arg lines stay visible.

|| "—".to_string(),
|ts| format!("{}s ago", (ts - r.ts).num_seconds().max(0)),
);
let tps = format::gen_tps_cell(r.gen_tps, r.gen_tps_observation.as_ref());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (UX): a held row prints 86*, but nothing in the panel explains the *. The other surfaces (instances table, Home, Observe, services manager) show format::HELD_LEGEND when a visible cell is held, and format.rs asks every held-marker surface to do so. Maybe render it when any shown row is held (and reserve that row in activity_layout).

let msg = if has_observed_throughput {
"no recent throughput — instance has been idle".to_string()
} else {
"no throughput data observed yet".to_string()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (UX): "observed yet" suggests data is on its way, but for engines that never get gen_tps wired up (managed non-vLLM) it stays that way forever. It can also sit right under a live gen: 100 tok/s in the summary when an instance was just discovered and history is still empty. Something neutral like "no throughput samples" avoids both.

Add a "recent throughput" panel to the instance detail modal that lists
the last few snapshot ticks where the instance had non-zero generation
throughput, with tok/s and TTFT/TPOT at that time (ROCMAI-196). The live
gen gauge mostly reads 0 tok/s between bursts, with nothing showing what
the last few serving runs looked like.

Built from the already-collected rolling snapshot history: idle ticks are
filtered out, held ticks collapse to the scrape's own tick, and the newest
five are kept. The panel is sized together with its row count and leaves
the launch args / env body at least max(6, half the modal) rows, collapsing
on shorter terminals. A `*` held row carries the shared held legend in the
panel title. Covered by scenario dash-42 and render tests in
ui/instance_detail.rs.

Signed-off-by: juhovainio <juho.vainio@amd.com>
@juhovainio

Copy link
Copy Markdown
Collaborator Author

Addressed review 5783507958 in 44b9e2d (history is now a single commit rebased onto current main, which resolves the conflict).

Blocking

  • Body starvation: the body now keeps max(6, inner_h / 2) rows (body_floor); the panel shrinks or collapses to fit. At 80x24 the panel shows one row and ≥5 launch-arg lines stay visible; new test detail_modal_at_80x24_keeps_several_launch_args_visible_beside_activity covers it (it fails if the floor goes back to 3). The 90x18 test now asserts the panel collapses and the scrollbar still renders.

Non-blocking

  • Held * legend: the panel title now appends format::HELD_LEGEND when a shown row is held (title rather than a reserved row, to keep the layout arithmetic unchanged); test covers held vs. fresh.
  • Wording: "no throughput data observed yet" is now "no throughput samples".
  • PR description updated (dash-42, tags, test command).
  • Test nits: timestamps in the 90x18 / 160x48 tests now run oldest→newest so 105 is the newest row; added a test for the .max(0) age clamp.

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @juhovainio, nice follow-up. I checked each point from my last round against this head and all are resolved: the body floor now keeps the launch args / env panes readable (the layout no longer starves them at the sizes I traced), the held * legend is shown in the panel title, the placeholder wording is neutral, the test fixtures run oldest to newest, the .max(0) clamp has a test, and the PR text now matches the scenario id and tags. CI is green on this head, and I found no blocking issues.

Non-blocking suggestions are inline (held TTFT/TPOT marker and reuse of latency_cell, test geometry vs. the real 80x24 body, silent collapse at 80x24, legend dropped on narrow terminals, the "idle" wording). Small extras:

  • The PR text says "at 80x24 the panel shows one row"; in the real app (body is 20 rows) it is hidden when there is data, so that sentence could be adjusted.
  • detail_modal_empty_history_on_short_terminal_... asserts the absence of "no throughput data", but the placeholder is now "no throughput samples", so that half of the assertion can't fail.
  • body_floor's .max(6) has no effect (below 12 rows the panel already collapses), so the doc line "keeps at least 6 rows" isn't strictly true.

🤖 by agenthub on AMD AgentHub

|ts| format!("{}s ago", (ts - r.ts).num_seconds().max(0)),
);
let tps = format::gen_tps_cell(r.gen_tps, r.gen_tps_observation.as_ref());
let ttft = format::ms_opt(r.ttft_ms);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (UX/consistency): TTFT/TPOT come from their own trackers, so a row with a fresh tok/s can carry a held latency from an earlier scrape, and here it is printed with no * (and the title legend only looks at the tok/s observation). The Instances table handles this with format::latency_cell(ms, observation), which marks held values and feeds the legend. Carrying ttft_observation/tpot_observation in ActivityRow and using latency_cell would fix it, and then ms_opt (which is the observation-less subset of latency_cell) can go. The test fixture for the Held tick ("carries no latencies of its own") is also something the runner doesn't actually produce.

.unwrap();
state.history.push_back(snap);
}
let mut term = Terminal::new(TestBackend::new(80, 24)).unwrap();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (test fidelity): draw_detail is called with the app's body rect (terminal minus 3 header rows and 1 footer row), but this test, and the 90x18 / 100x28 ones, pass f.area(), so each is 4 rows taller than the terminal size in its name. At a real 80x24 the body is 20 rows, inner is 15, avail is 3, and a data panel (min 4 rows) collapses, so contains("recent throughput") would not hold there. Driving these through ui::draw, or passing Rect of height rows - 4, would make them match what users see. The PR text ("at 80x24 the panel shows one row") has the same off-by-4.

/// handed rows it can't draw. A panel too short for its header plus one row
/// (or the placeholder line) collapses to `(0, 0)` so the body gets the rows.
fn activity_layout(n_rows: usize, inner_h: u16) -> (u16, usize) {
let avail = inner_h.saturating_sub(4 + 1 + body_floor(inner_h));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (UX): the body floor fixes the starvation, thanks. One side effect at the real 80x24 default: a panel with data needs 4 rows and avail is 3, so it disappears silently, while the 3-row empty placeholder still fits and takes 3 of the 7 launch-arg rows (7 -> 4, so env vars start to need scrolling). The panel is hidden where it has the most to say and shown where it has nothing. Collapsing the placeholder at the same threshold as the data panel (or a one-line hint like "recent throughput hidden - enlarge terminal") would keep that consistent.

.is_some_and(|m| m.freshness == rocm_dash_core::metrics::ObservationFreshness::Held)
}) {
title.push_str(" · ");
title.push_str(format::HELD_LEGEND);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (UX): panel::bento drops a title that doesn't fit (title_fits), and the longest form here ("recent throughput (newest N) · * = held (prior scrape window)") is ~61 chars, so on terminals narrower than about 80 columns the label, "(newest N)" and the legend all vanish together, leaving * rows unexplained. Picking a shorter variant (e.g. * held) when it doesn't fit, or putting the legend in the body, would avoid that.

if inner.height == 0 {
return;
}
if rows.is_empty() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (UX): "instance has been idle" is chosen when any finite sample exists in the last ~240 ticks, not from the current state. If the observation has since expired, the scrape is failing, or the instance restarted under the same id, the summary shows gen: - / "unknown" while this says it's idle. Basing it on the latest tick's sample, or using neutral wording ("no non-zero throughput in recent history"), would avoid over-claiming.

@juhovainio
juhovainio added this pull request to the merge queue Nov 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 5, 2026
@juhovainio
juhovainio added this pull request to the merge queue Nov 5, 2026
Merged via the queue into main with commit 44b1a6f Nov 5, 2026
30 checks passed
@juhovainio
juhovainio deleted the jira-ROCMAI-196 branch November 5, 2026 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants