Repository navigation
ROCMAI-196: show recent serving activity in instance detail modal - #721
Conversation
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · d48d5cd
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Review — needs work
Full review of the whole change.
Implements part of ROCMAI-196, which asks for a rolling summary of recent serving runs (tokens, tok/s, latency). This change shows recent non-idle snapshot ticks instead, and the description says so. It shows no token counts.
Blocking
-
[new] The panel cuts off its oldest row whenever it has 2 or more rows —
crates/rocm-dash-tui/src/ui/tabs/instances.rs(activity_hcomputation indraw_detail;render_recent_activity)
activity_h = 2 + 1 + Ncounts the two borders, the header and N rows. It leaves out the top padding row thatpanel::bentoadds once the box is at least 5 rows tall (padding_for:top = u16::from(area.height >= 5),crates/rocm-dash-tui/src/ui/panel.rs). From 2 rows upward the table loses one line, and it is the oldest entry. A probe test renderingdraw_detailwith 5 active ticks at 120x36 and at 160x48 showed only 4 of the 5 rows (104…101 tok/s, with a blank line above the header). With 2 ticks it showed 1. With 1 tick (box height 4, no padding) the row appeared. SoRECENT_ACTIVITY_CAP = 5really shows 4 rows. The only rendering test with rows uses a single tick, which is why it passes.
Confidence 100 · logic · Fix: add the bento top padding toactivity_h(or work the height out frompadding_for), and assert in a render test that all N rows appear for N ≥ 2. -
[new] "no recent activity — instance has been idle" appears even when the dashboard has no throughput data —
instances.rsrender_recent_activityplaceholder; testdetail_modal_shows_idle_placeholder_when_no_recent_activity
The list is empty whenever no tick has a finitegen_tps > 0. That happens in several cases where nothing is known to be idle:gen_tpsisNoneuntil two scrapes exist.gen_tpsgoes back toNoneafter the validity window.- It is never set for managed non-vLLM engines.
crates/rocm-dash-daemon/src/registry.rsdiscover_managed_services: "gen_tps for managed non-vLLM engines (e.g. Lemonade) is not yet wired". A Lemonade instance that is serving would always be shown as idle. - History is empty right after start and after a replay seek (
reset_for_seekclears it).
The new test pins this. It builds the instance with
mk_inst_obs("idle-detail", None, None)(gen_tps = None, empty history) and asserts the idle message. AGENTS.md §3 says a message about what happened must be asserted together with the behaviour, and "a pin over a false claim holds the claim in place". The live gauge in the same modal already saysgen freshness: unknownin this state.
Confidence 85 · logic · Fix: say "idle" only when the history window holds realSome(0.0)samples. Otherwise say "no throughput data". Preferably state the window too (history holds at mostHISTORY_CAP= 240 one-second ticks). -
[new] Held (carried-forward) samples appear as separate activity rows with no held marker —
instances.rsrecent_activity_rows,render_recent_activity(usesformat::tps_opt)
Snapshots arrive every 1 s (gpu_tick), but instances are scraped every 2 s (instance_tick,crates/rocm-dash-daemon/src/runner.rs). Ticks between scrapes carrygen_tpsforward withObservationFreshness::Held. On top of that,gen_tpsis a rolling rate overrate_window, at least 10 s (crates/rocm-dash-core/src/observation.rs). So one short burst fills all the rows with consecutive "0s/1s/2s… ago" entries, about half of which are copies of the previous scrape.
Every othergen_tpssurface in this file renders throughformat::gen_tps_cell, which appendsHELD_MARKERfor held values (instances.rsline ~140, plushome.rs,pane.rsandserving.rs). This panel usesformat::tps_optand drops the marker, which breaks the held-marker convention forgen_tpssurfaces.
Confidence 85 · logic · Fix: keep onlyFreshobservations, or group consecutive non-zero ticks into runs. Grouping into runs is also closer to the ticket. If held values stay, render them throughgen_tps_cell. -
[new] The fix commit d48d5cd (the
BODY_FLOORcap) has no regression test —instances.rslet activity_h = activity_h.min(inner.height.saturating_sub(4 + 1 + BODY_FLOOR));
Deleting this line left all 50tabs::instancestests green (mutation M1,cargo test -p rocm-dash-tui --lib -- tabs::instances: 50 passed, 0 failed). A probe showed the cap does matter: at 90x16 and 80x12 the scrollbar is drawn with the cap and missing without it. AGENTS.md §3 requires every bug fix to ship with a regression test that fails before the fix.
Confidence 100 · logic · Fix: add a short-terminal render test with overflowinglaunch_argsand 5 activity rows, and assert the scrollbar is drawn. -
[new] No Gherkin scenario for the new panel, and no stated reason —
tests/e2e-cucumber/features/dash.feature
The detail modal already has PTY scenarios:@id:dash-instance-detail-dims-backdropand@id:dash-instance-detail-scroll-hint, with steps indash_steps.rs/tui_driver.rs. The panel is new user-visible output in that modal, and its height (3–8 rows) now decides whether the body overflows. The geometry notes intui_driver.rsreason only about the args/env panes. AGENTS.md §3 requires a scenario, or a sentence in the PR explaining why there is none. The PR has neither.
Confidence 85 · architectural · Fix: add a scenario that checks the panel's rows (and the no-data vs idle wording), or justify its absence in the PR text.
Non-blocking
- The non-finite filter is untested —
recent_activity_rowsv.is_finite() && v > 0.0. Droppingis_finite()left every test green (mutation M6). ASome(f64::INFINITY)tick would show up as a row. Add one NaN/Inf tick to the existing selection test. Confidence 100 · logic. - The TTFT/TPOT formatting closures copy the instances table's own (
.filter(is_finite).map_or_else(dash, "{v:.0}ms"), the same pattern near line 143). Ages are shown as raw seconds ("239s ago") even thoughformat::durationexists. AGENTS.md §6 prefers shared helpers. Aformat::ms_optwould remove both copies. Confidence 80 · mechanical. - The Home tab already has a "recent activity" panel with a different meaning (
home.rs: jobs/actions, and "no recent activity — serve a model…"). Reusing the title for throughput samples is confusing. Confidence 70 · mechanical. - The description says "Marking as draft", but the PR is not a draft (
isDraft: false). The approximation it says it wants feedback on is the open question in the Blocking items above. Confidence 100 · mechanical.
Decisions for the author
- Per-tick samples versus grouped runs — tradeoff. The description says plainly that there is no per-request telemetry. But consecutive ticks can be grouped into runs (start age, duration, peak/avg tok/s, tokens ≈ ∫tok/s) from the same history, which would get much closer to the ticket's "summary of the last few serving runs". Per-tick rows are simpler, but with held samples and the ≥10 s rate window they mostly show one burst fading out.
- Vertical space — tradeoff. Once it has data the panel takes 8 rows. That costs the body about 40% of its height at the 120x36 popup cap, and most of it at 80x24, and the body box resizes as samples come and go. A one-line summary, collapsing the panel when there is no activity, or placing it beside the summary would keep
launch_args/env_varsreadable.
Positive signals
- Separating the pure
recent_activity_rowsfrom the renderer makes the selection rules testable. The tests for filtering, ordering and the cap catch mutations M2, M3 and M4.
Deployment notes
None
What this covered
Read crates/rocm-dash-tui/src/ui/tabs/instances.rs in full (the only changed file), plus panel.rs (bento/padding_for/scrollbar), format.rs, the history handling in app/mod.rs, and the producer side in rocm-dash-core/observation.rs and rocm-dash-daemon (runner.rs, registry.rs, server.rs). Diff prw-base...HEAD = 32834b29…d48d5cd9 (merge base; prw-base is at ea1d8713).
Verdicts for the added tests, all run with cargo test -p rocm-dash-tui --lib -- tabs::instances (baseline 50 passed):
| Test | (1) Passes with the change reverted? | (2) Mutations it catches / misses |
|---|---|---|
| skips_idle/newest_first | No (the function would not exist) | Catches >=0 (M2) and dropped .rev() (M3). Misses removal of is_finite (M6). |
| caps_to_requested_len | No | Catches dropped .take (M4). |
| idle_placeholder | No | Catches the placeholder mutation (M5). Asserts the false-idle case (Blocking). |
| shows_recent_activity_row | No | Catches the unwired renderer (M7), constant age (M8) and zero-height layout (M9). Misses the clipping of the oldest row because it uses a single tick. |
| (none) | — | The BODY_FLOOR cap (M1) survives every test. |
CI at d48d5cd: everything green except "E2E tests". Its two unexpected failures (dash-gen-tps-expiry-boundary, dash-esc-menu-quit-row-confirms-when-serving, both quit timeouts in Observe-view scenarios that never open the detail modal) also fail on main's own CI run at ea1d8713 (run 39793631227). So they are not attributed to this change. GPU E2E lanes were skipped.
Fan-out ran 2 workers, as the caller's budget allowed. The design/rules/history worker returned and its claims were re-checked against source. The code-pass worker had not returned by the time limit and could not be stopped from here. I used its raw mutation and probe cargo logs, written under the checkout's .prw-scratch/, after checking the mutation script and the outputs myself. Its own judgement was not received, so the code pass is partly not independent.
Did not run: the prior-changes pass (earlier PR review comments are out of bounds here); the Windows-target check; the e2e suite locally.
ROCMAI-196 was retrieved by its identifier (summary, description and status only).
Not reconciled against any PR discussion here.
Addresses rominf's review (5625116225) on PR #721: - Fix activity_h to account for panel::bento's top-padding row once the box reaches 5 rows tall, so RECENT_ACTIVITY_CAP = 5 actually shows all 5 rows instead of clipping the oldest one. - Split the empty-history placeholder into "has been idle" (a real Some(0.0) sample was observed) versus "no throughput data" (never observed, past the observation's validity window, a managed non-vLLM engine, or a freshly-seeked replay) so the message can't claim idle when nothing is actually known. - Render recent-activity tok/s through format::gen_tps_cell so held (carried-forward) samples get the same held marker as every other gen_tps surface in this file. - Add a short-terminal regression test for the BODY_FLOOR cap itself (overflowing launch_args + a full activity panel), proving the scrollbar thumb survives. - Cover a non-finite (NaN/Inf) tick in the row-selection test. - Add format::ms_opt to de-duplicate the TTFT/TPOT formatting that was copy-pasted between draw_instances_table and render_recent_activity. - Rename the panel "recent activity" -> "recent throughput" so it isn't mistaken for the Home tab's unrelated "Activity * node" panel. No new Gherkin scenario was added for the panel: demo mode's gen_tps oscillates continuously and is always positive with gen_tps_observation: None, so the PTY-driven E2E framework can't deterministically exercise the idle-vs-no-data distinction without substantial new step-definition plumbing. Existing unit/render tests already cover both the selection rules and both placeholder messages. Signed-off-by: Juho Vainio <juho.vainio@amd.com>
|
Addressed the review from @rominf (pullrequestreview-5625116225) in 4937252. Blocking items
Non-blocking
Decisions I'm leaving to you (not changed)
Build/test/lint: |
Addressed in 4937252 — see PR comment for the full mapping of changes to review items.
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 4937252
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Review — needs work
Full review of the whole change.
Implements part of ROCMAI-196: the ticket asks for "a rolling summary of recent serving runs (tokens, tok/s, latency)". This change shows recent non-idle snapshot ticks with tok/s, TTFT and TPOT, and no token counts. The description says so itself, and the PR does not claim to close the ticket.
Round 2. The earlier change request (review 5625116225, at d48d5cd) was dismissed by the author. Its objections were checked again against 4937252; see Earlier objections at the end. Each finding is marked [new] or [standing].
Blocking
-
[new] "no throughput data in the last 240s" is often untrue —
crates/rocm-dash-tui/src/ui/tabs/instances.rs:790HISTORY_CAPcounts snapshots, not seconds. Its doc atapp/mod.rs:62says "How many snapshots to keep".- The daemon sends one snapshot per
gpu_tick(runner.rsrun_loop). That tick is configurable: the config tests accept 0.5s, which makes the window 120s, and a 2s tick makes it 480s. - History is often much shorter than 240 entries:
- it starts with a single snapshot when the dash opens;
- a late-joining client gets at most a 30-snapshot backlog;
reset_for_seekclears it entirely on a replay seek.
- In each of those cases the modal claims a 240-second observation window it never had. The doc comment on
has_observed_throughputitself lists "history freshly cleared by a replay seek" as one of the cases that reach this message. - This breaks AGENTS.md §3 ("A message about the CLI's own behavior is asserted together with the behavior"):
detail_modal_shows_no_data_placeholder_…checks only the substring "no throughput data", so the false part is never asserted. - The previous round of this automation suggested stating the window and described the history as "240 one-second ticks". That description was wrong in the same way, so this part of the problem comes from our own earlier suggestion.
- Confidence 85 · logic · Fix: derive the window from the history itself (newest timestamp minus oldest), or drop the number. Assert the rendered window against the state the test sets up.
-
[new] On a short terminal the panel shows only its column header, which looks like an empty table —
instances.rs:737(theactivity_hcap),instances.rs:~2650(the test that pins this geometry)- Rendering
detail_modal_body_scrollbar_survives_a_full_activity_panel_on_a_short_terminalat 90x16 with an output dump showsrecent throughputwith onlyTIME TOK/S TTFT TPOTand no data rows, even though history holds 5 bursts. - When the cap lands at 1–2 rows, nothing renders at all, so the idle/no-data placeholder also disappears without notice.
- The new test passes in exactly this state, so it pins it.
- Confidence 100 · logic · Fix: collapse the panel to height 0 (or one summary line) when it cannot fit the header plus at least one row. Make the short-terminal test assert whatever is shown.
- Rendering
-
[standing, re-found this round] No Gherkin scenario covers the new user-visible panel or its two placeholder messages —
tests/e2e-cucumber/features/dash.feature(no change)- AGENTS.md §3: "User-observable behavior needs a scenario, not only a unit test… a unit test asserting the internal helper does NOT discharge this."
- The instance detail modal already has scenarios (dash-13
@id:dash-instance-detail-dims-backdrop, dash-22@id:dash-instance-detail-scroll-hint). Neither asserts the panel. - Confidence 85 · convention · Fix: extend dash-13/dash-22 or add a scenario that asserts the panel, or justify its absence in the PR text.
- Response to the author's reason for not adding one. We weighed the reason given in the review-reply comment against the code, after the review above was finished. It does not hold:
- The PTY harness does not only drive demo mode.
@id:dash-gen-tps-held-after-scrape-failureand@id:dash-gen-tps-expiry-boundary(dash.feature:74-113) run a real daemon against "a managed model exposes scripted serving metrics". They pin observation time and inject "the metrics endpoint fails transiently". So the populated panel, the held*marker and expiry back to the no-data wording can all be reached by reusing that setup. The idle case would need a counter that stays flat; that is likely possible but was not checked. - The point about demo mode is correct:
demo.rsalways reports positivegen_tpswith no observation. That only limits scenarios built on demo mode. - §3 exempts only purely internal changes, and gated lanes only when the PR names them. "The unit/render tests already cover it" is the argument §3 rules out explicitly. The reason is also given in a comment, not in the PR text.
- This weighing was done by a reviewer that had been shown the objection, so it counts as anchored evidence. The blind pass found the gap on its own.
- The PTY harness does not only drive demo mode.
-
[new] Two branches the change adds go green with no test catching them —
instances.rs:764(activity row cells),instances.rs:379(has_observed_throughputis_finite)- Each branch was mutated and
cargo test -p rocm-dash-tui --lib -- ui::was run (543 tests, the new tests included). Both mutations passed everything:- swapping the activity row's TTFT and TPOT cells (
[time, tps, tpot, ttft]); - replacing
is_some_and(f64::is_finite)withis_some().
- swapping the activity row's TTFT and TPOT cells (
- Nothing asserts the TTFT/TPOT columns of the new rows. The second mutation means a history of only non-finite samples would print "has been idle".
- Verdicts for the added tests:
- All 8 fail with the production change reverted, because they call the new symbols.
- Removing the
top_padreservation fails…all_activity_rows_up_to_the_cap. - Removing the
BODY_FLOORcap fails…body_scrollbar_survives…. - Dropping
ms_opt's finite filter fails the existingnonfinite_ttft_tpot_render_dash_never_nan. - The two mutations above survive.
- Confidence 100 · test · Fix: assert distinct TTFT/TPOT values in an activity-row render test, and add a history containing only
Some(INFINITY)to the no-data test. - This change request is not filed because of this finding alone. The previous round rated the same kind of gap (an untested non-finite filter) as non-blocking. Treat the first three findings as what needs to change.
- Each branch was mutated and
Non-blocking
- [new] Code comments describe something the code does not do —
instances.rs:338-343(recent_activity_rowsdoc),instances.rs:~762(render_recent_activitydoc)- The comments say the panel is "the 'rolling summary of recent serving runs' from ROCMAI-196" and that "only the spikes are worth surfacing".
- The code returns the 5 newest non-zero snapshot ticks, and the PR description itself says "recent throughput here means recent non-idle snapshot ticks rather than discrete serving runs".
- The description governs, because it matches the code path. Fix the comments.
- Confidence 85 · documentation.
Decisions for the author
- The 5 rows almost always show the last ~5 seconds of a single burst, often with repeated values — answered-by-intent
- PR description: "'recent throughput' here means recent non-idle snapshot ticks rather than discrete serving runs … happy to adjust the granularity."
- That covers the granularity. What it does not state is the consequence:
- Each
gen_tpssample is re-sent on every snapshot between scrapes (runner.rs:914; the default 1sgpu_tickagainst a 2sinstance_tick). - The rate is averaged over at least 10s (
observation.rsrate_window). - So one burst keeps
gen_tps > 0for 10 or more consecutive ticks. The 5-row panel then shows "0s…4s ago" of that one burst with duplicated or smoothed values, and never an earlier separate spike. - On a continuously busy instance it mirrors the live gauge.
- The TIME column is the snapshot's age, not
gen_tps_observation.observed_at.
- Each
- Without the description this would have been blocking (the panel cannot meet the ticket's "last few serving runs").
- Confirm this is the intended shape. Otherwise, dedupe by
observed_ator group consecutive non-zero ticks into runs.
Positive signals
- Reserving the activity panel's height through
panel::padding_forinstead of re-deriving bento's padding threshold. A mutation check confirms the regression test guards it. - Moving the TTFT/TPOT formatting into
format::ms_optkeeps the instances table and the new panel on one formatter. The existing non-finite test now covers both.
Deployment notes
None
What this covered
- Read: both changed files completely, against
prw-base(1c33248…) through 4937252, plus the surrounding code:panel.rs(bento,padding_for, scrollbar);app/mod.rshistory handling and replayreset_for_seek;- the daemon's snapshot and observation cadence in
runner.rsandobservation.rs; dash.feature.
- Fan-out: one independent read-only worker ran the code pass. The agent-instruction, history, code-comment and design passes were run in the reviewing context, so they were not independent. The design questions were not split out to separate workers; the 2-worker budget was respected.
- Runs:
cargo test -p rocm-dash-tui --lib -- tabs::instances: 54 passed, the 8 new tests included.- Five single-branch mutations under
-- ui::(543 tests). - One render dump of the short-terminal test.
- All of these used the checkout's own
target/.
- CI: hosted CI for this head passed. The self-hosted GPU E2E lanes were skipped.
- Did not run: the prior-changes pass, because earlier reviews and PR comments are out of bounds for the blind review.
- Ticket: ROCMAI-196 was retrieved by its identifier (summary, description, status), without comments.
- The findings above were reached blind. The author's review-reply comment was weighed only afterwards, and only for the objections below.
Earlier objections (from review 5625116225)
Each one was checked by putting its wording and the author's reply to the reviewer after the blind review was finished. Those verdicts are anchored evidence.
- The oldest row was cut off — settled.
activity_hnow reservespadding_for(..).top. A mutation removing it failsdetail_modal_shows_all_activity_rows_up_to_the_cap. - "Instance has been idle" was shown when there was no throughput data — settled.
has_observed_throughputrequires a finite sample in history, so empty or all-Nonehistories (after startup, a replay seek, or the validity window) now get the no-data wording. The test that pinned the old message was replaced.- Observation, not a finding: if a single
Some(0.0)sample is followed only byNone, the panel still says "has been idle". - Not checked: whether the Lemonade collector ever reports
Some(0.0).
- Observation, not a finding: if a single
- Held samples had no held marker — settled. The TOK/S cell uses
format::gen_tps_cell, andobservation.rssnapshot()tags carried-forward ticks between scrapes asHeld(runner.rs:914-924), so those rows show*. Keeping one row per tick is covered under Decisions for the author. - The
BODY_FLOORcap had no test — settled. A mutation removing the cap failsdetail_modal_body_scrollbar_survives_a_full_activity_panel_on_a_short_terminal. That test pins a header-only panel, which is the new finding above. - No Gherkin scenario — still holds; see Blocking.
|
Addressed review 5630756475 in cd6bfbf. Blocking
Non-blocking
Left unchanged (decision for the author)
Checks: |
Addresses rominf's review (5625116225) on PR #721: - Fix activity_h to account for panel::bento's top-padding row once the box reaches 5 rows tall, so RECENT_ACTIVITY_CAP = 5 actually shows all 5 rows instead of clipping the oldest one. - Split the empty-history placeholder into "has been idle" (a real Some(0.0) sample was observed) versus "no throughput data" (never observed, past the observation's validity window, a managed non-vLLM engine, or a freshly-seeked replay) so the message can't claim idle when nothing is actually known. - Render recent-activity tok/s through format::gen_tps_cell so held (carried-forward) samples get the same held marker as every other gen_tps surface in this file. - Add a short-terminal regression test for the BODY_FLOOR cap itself (overflowing launch_args + a full activity panel), proving the scrollbar thumb survives. - Cover a non-finite (NaN/Inf) tick in the row-selection test. - Add format::ms_opt to de-duplicate the TTFT/TPOT formatting that was copy-pasted between draw_instances_table and render_recent_activity. - Rename the panel "recent activity" -> "recent throughput" so it isn't mistaken for the Home tab's unrelated "Activity * node" panel. No new Gherkin scenario was added for the panel: demo mode's gen_tps oscillates continuously and is always positive with gen_tps_observation: None, so the PTY-driven E2E framework can't deterministically exercise the idle-vs-no-data distinction without substantial new step-definition plumbing. Existing unit/render tests already cover both the selection rules and both placeholder messages. Signed-off-by: Juho Vainio <juho.vainio@amd.com>
cd6bfbf to
a87d1a8
Compare
nowycondro
left a comment
There was a problem hiding this comment.
Automated review pass (recall-biased). Three findings, most-severe first: a test-assertion gap in the new e2e step that doesn't actually prove the "recent throughput" panel rendered populated rows, plus two cleanup items (a redundant double scan of history, and some fragile inline panel-sizing arithmetic worth generalizing). No crash-level correctness bugs found in the new rendering/layout logic — traced the height-reservation arithmetic by hand and independently confirmed it's self-consistent (bento always recomputes its own padding from the final area it's given, so the pre-computed top_pad can't desync from what's actually rendered).
| |screen| { | ||
| screen.contains("recent throughput") | ||
| && screen.contains(" ago") | ||
| && !screen.contains("no throughput data") |
There was a problem hiding this comment.
Correctness (test coverage): this assertion doesn't scope to the "recent throughput" panel, so it can pass without the panel ever showing a populated row.
Two gaps:
- The negative guard only excludes
"no throughput data"— the "no data observed" placeholder — but never excludes"no recent throughput — instance has been idle", the other placeholderrender_recent_activitycan render (instances.rs). If the panel fell back to the idle placeholder instead of real rows, this guard alone wouldn't catch it. screen.contains(" ago")isn't scoped to the activity table. In the same modal,render_summary's freshness line (format::gen_tps_detail_freshness) renders"held · {age_s}s ago"for aHeldobservation — which also contains" ago". So the check could pass purely from that summary line while the activity table itself shows the wrong placeholder.
Verified this doesn't currently misfire only because the matching scenario (MockServer::start_with_scripted_metrics) stays in "Growing" mode, which never produces a Held observation — so today it happens to pass for the right reason. But the assertion as written doesn't actually guarantee that; a future change to the scripted-metrics mode, or reuse of this step for another scenario, could make it pass while the real feature is broken. Consider requiring the row's own "Ns ago" shape (anchored near a TOK/S/TTFT row) and also excluding "has been idle".
| // | recent throughput (recent non-idle ticks, ROCMAI-196) | body (min) | footer (1) | ||
| let snap_ts = state.latest.as_ref().map(|s| s.timestamp); | ||
| let activity = recent_activity_rows(&state.history, &inst.container_id, RECENT_ACTIVITY_CAP); | ||
| let has_observed_throughput = has_observed_throughput(&state.history, &inst.container_id); |
There was a problem hiding this comment.
Efficiency: redundant full scan of state.history on every detail-modal render.
recent_activity_rows (line 717) and has_observed_throughput (line 718) each independently iterate state.history (up to HISTORY_CAP = 240 entries, from app/mod.rs) looking for this instance's samples. The worst case — exactly when has_observed_throughput's result is actually needed (activity.is_empty(), i.e. no non-idle tick found) — is also the case where recent_activity_rows was forced to scan the entire history without a match (since .take(cap) never short-circuited), so has_observed_throughput then does a second full 240-entry scan right after. This runs on every redraw of the detail modal (every key press/tick while it's open).
Consider merging into one pass, e.g. have recent_activity_rows also return (or an accompanying helper compute in the same iteration) whether any finite sample was ever seen, so the empty-case doesn't require re-scanning the same data twice.
| // rows tall (`padding_for`). Reuse that helper instead of re-deriving the | ||
| // threshold, or this reservation silently drifts out of sync with it | ||
| // again — which is exactly how the oldest activity row got clipped. | ||
| let top_pad = panel::padding_for(Rect::new(0, 0, 1, raw_h)).top; |
There was a problem hiding this comment.
Altitude/simplification: this panel re-derives bento's padding/sizing rules inline instead of generalizing a shared helper.
draw_detail now computes raw_h → top_pad (by constructing a throwaway Rect just to probe panel::padding_for) → caps against BODY_FLOOR → collapses below min_h — four interdependent, hand-tuned steps, duplicated nowhere else, to answer "how tall should this bento-boxed table be, accounting for its own adaptive padding, without starving its sibling panel." The comments on lines 726-729 explicitly call out that this is the second time the padding threshold had to be re-synced by hand ("which is exactly how the oldest activity row got clipped").
This is self-consistent today (verified: panel::bento always recomputes its real padding from whatever area it's actually given, so there's no live bug), but the fragility the comments describe is real: the next variable-height bento panel added to this codebase will either duplicate this whole dance again or silently reintroduce the clipping bug the comments warn about. Consider pushing a panel::reserved_height(content_rows, floor, available)-style helper into panel.rs (next to padding_for) so the padding-aware sizing logic has one home instead of being re-derived per call site.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · cd6bfbf
Reviewed the diff at head cd6bfbfae878b5853ca5a376c5ab420272bec71c (branch jira-ROCMAI-196), focused on crates/rocm-dash-tui/src/ui/tabs/instances.rs and the new e2e step in tests/e2e-cucumber/tests/e2e/dash_steps.rs, plus supporting context in panel.rs, format.rs, and app/mod.rs. No Rust toolchain was available in this sandbox, so nothing was compiled or executed; this is a static read of the diff, cross-checked by hand-tracing the bento padding/height-reservation arithmetic across terminal sizes and activity-row counts. I did not run the e2e/cucumber suite, did not verify GPU-backed lanes, and did not independently re-verify the prior rounds of review from rominf beyond skimming them for context (I formed my own judgment on the current head).
Findings (non-blocking only; no blocking/crash-level issues found):
-
(medium confidence) Weak e2e assertion —
tests/e2e-cucumber/tests/e2e/dash_steps.rs(new step around line 1645). The screen-match predicate used to confirm the "recent throughput" panel rendered populated rows only excludes the "no throughput data" placeholder, not the other placeholder ("no recent throughput — instance has been idle"), and the" ago"substring check isn't scoped to the panel —render_summary's freshness line can also render "… ago" for a Held observation elsewhere in the same modal. The scripted-metrics scenario exercised today happens to never produce a Held observation, so it passes for the right reason, but the assertion itself doesn't actually guarantee the panel rendered real activity rows rather than falling through to a placeholder. -
(low confidence, cleanup) Redundant double scan —
crates/rocm-dash-tui/src/ui/tabs/instances.rs(around line 718).recent_activity_rowsandhas_observed_throughputeach independently do a full linear scan ofstate.history(up toHISTORY_CAP= 240 entries) on every detail-modal render. When an instance has no non-idle tick in history, both scans run to completion back-to-back on every redraw of the open modal. Not a correctness bug, just avoidable repeated work that a single combined pass over history would eliminate. -
(low confidence, maintainability) Inline re-derivation of bento padding —
crates/rocm-dash-tui/src/ui/tabs/instances.rs(around line 730). The new panel re-derivespanel::bento's padding/sizing rules inline (raw_h/top_pad/BODY_FLOOR/min_h) rather than via a shared helper. I traced this by hand across the range of terminal sizes and row counts and it is self-consistent today —panel::bentoalways recomputes its own padding from the final area it's actually given, so there's no clipping bug currently — but this is exactly the kind of duplicated hand-tuned arithmetic the adjacent code comments say already caused a clipping regression once before. Worth factoring into a shared helper (e.g. areserved_height(...)function inpanel.rs) before the next variable-height bento panel is added.
Overall verdict: no blocking findings on the current diff. The height-reservation logic that prior review rounds (from rominf) flagged as broken appears to have been fixed and is now self-consistent by my independent trace. The above are suggestions for tightening test coverage and reducing future-maintenance risk, not correctness defects.
🤖 Generated with Claude Code
|
🤖 Automated review (Claude Code) · a87d1a8 I did an independent fresh pass over the current diff (recent-activity/throughput panel in Possible correctness issues (medium confidence, worth a look): (1) Non-blocking: the new e2e step's screen-predicate (dash_steps.rs:~1636) checks for I did not have a Rust toolchain set up to build or run the crate/e2e tests in this sandbox, so all of the above is from static reading of the diff and surrounding code, not from execution. 🤖 Generated with Claude Code |
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · a87d1a8
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Review — needs work
Full review of the whole change.
Partly implements ROCMAI-196, which asks for a "rolling summary of recent serving runs (tokens, tok/s, latency)". This change shows tok/s and latency for recent non-idle snapshot ticks. It does not group ticks into runs and has no token counts. The code comment says the coverage is partial, and the PR does not claim to close the ticket.
Round 3. The earlier change request (review 5630756475, at 4937252) was dismissed; its objections were checked again against a87d1a8 and are settled at the end. Each finding is marked [new] or [standing].
What this change request is filed for: the first finding below only. The other two items under Blocking do not gate on their own:
- the description mismatch is in the PR description, and the squash commit message is what lands on main;
- the same
E2E tests (MI300X, current)scenarios (chat-06, serve-01, serve-02, serve-07) also fail on main's latest run of that lane, and this PR touches only dash-tui and its e2e files.
Blocking
- [new; the part of the standing §3 objection that still holds] The new e2e step passes when the panel shows the idle placeholder instead of any samples. Locations:
tests/e2e-cucumber/tests/e2e/dash_steps.rs:1634-1651,crates/rocm-dash-tui/src/ui/format.rs:317.- The step passes if the screen contains "recent throughput" and " ago", and does not contain "no throughput data".
- "recent throughput" is in the panel title and also in the idle placeholder text ("no recent throughput — instance has been idle").
- " ago" also comes from the summary's freshness line, which
gen_tps_detail_freshnessrenders asheld · Ns ago. - In this scenario the daemon publishes a snapshot every 1 s but scrapes every 2 s, so every other frame is Held.
- I confirmed it by running a render probe at 120x40: Held observation, history holding only
Some(0.0). The screen showedgen freshness: held · 1s agoandno recent throughput — instance has been idle, and the step's predicate evaluated true. - The step's own comment says the placeholders lack " ago". That is true of the panel but not of the screen.
- So "lists recent throughput samples" is a check that cannot fail on the "no samples" case it is named for.
- Confidence 100 · logic · Fix: match a row-shaped line, e.g.
\d+s agofollowed by a numeric tok/s cell on the same line, or also rejecthas been idle. Correct the comment.
- [new] The PR description misstates the change.
- It says the placeholder distinguishes "no throughput data in the last
HISTORY_CAPs". Commit a87d1a8 removed that wording; the code now renders "no throughput data observed yet" (instances.rs,render_recent_activity). - It does not mention the new e2e scenario dash-38 (
dash.feature, new step indash_steps.rs). - It does not mention that
draw_table's TTFT/TPOT cells now go through the newformat::ms_opt. Output is unchanged; both render—. - Its test plan lists only the unit tests.
- Confidence 85 · mechanical · Fix: update the description to match the code. The work itself is fine.
- It says the placeholder distinguishes "no throughput data in the last
- [new] One required CI check is failing:
E2E tests (MI300X, current).- The run had 6 failed scenarios. Reconciliation counted 5 as expected xfails and 1 unexpected regression:
therock-next-09-live-install-reports-vllm-rocm10x-discovery-pins, whererocm engines install vllm --reinstall --yesfailed with rc=1. - This PR touches only dash-tui and its e2e files. dash-38 passed on that lane.
- Confidence 85 · architectural · Fix: compare against main's run for that lane; if it is unrelated, re-run it or escalate.
- Caller's comparison: main's latest MI300X run fails the same scenarios (chat-06, serve-01, serve-02, serve-07), and this run's serve-07 failure is a llama.cpp/TheRock download failure. This does not gate.
- The run had 6 failed scenarios. Reconciliation counted 5 as expected xfails and 1 unexpected regression:
Non-blocking
- [new] Held ticks become duplicate activity rows, and their age is measured from the wrong time. Locations:
instances.rs:329,:358,:812.GenTpsTracker::snapshot(rocm-dash-core/src/observation.rs) repeats the same value as Held on every tick until the observation's validity window expires.recent_activity_rowskeeps every non-zero tick, so one scrape can fill several of the 5 rows (e.g.86*three times) and push out distinct samples.- TIME is computed from
snap.timestamp, notobserved_at. A held row's "Ns ago" therefore disagrees with the summary'sheld · Ns ago. - Confidence 70 (read, not run) · logic · Fix: dedupe rows by
observed_ator skip Held ticks, and take the age fromobserved_atwhen it is present.
- [new] One assertion can never fail. Location:
instances.rs:2523,!out.contains("240").- No production string contains "240" since a87d1a8 removed the window wording, so its message ("must not claim an observation window…") pins nothing.
- Confidence 90 · mechanical · Fix: delete it.
- [new] The render-test fixture does not match how production fills history. Location:
state_with_snap(instances.rs:1992).- The fixture sets
latestbut never pushes it intohistory. Production'spush_snapshot(app/mod.rs:1488) always does both. - So no test covers the real shape: a currently-generating instance gets a
0s agofirst row that repeats the live gauge. Also, a liveSome(0.0)with no older history reads "idle" in production but "no data" in the tests. - Confidence 75 · logic · Fix: add one test that pushes the latest snapshot into history, and decide whether the live tick belongs in "recent".
- The fixture sets
- [new] The scenario's comment gives the wrong reason for not using the demo. Location:
dash.feature:643-649.- It says demo instances report "a constant positive gen_tps" that the demo replay "cannot exercise".
- In fact demo gen_tps oscillates and is always ≥ 90 (
rocm-dash-daemon/src/demo.rs:595-597), and those snapshots go intohistory. So--demowould fill the panel with rows. What the demo cannot reach is the idle vs no-data split. - Confidence 80 · mechanical · Fix: correct the comment.
- [new]
.max(0)on the row age is untested. Location:instances.rs:812.- Removing it left all 17 related tests green (mutation run).
- It is defensive: history always ends with the latest snapshot, so a negative age needs a clock anomaly.
- Confidence 70 · mechanical · Fix: add a row timestamped after
snap_ts, or accept it as defensive.
- [new] Commit bodies contradict the final tree. a87d1a8 says "Add dash-37 PTY scenario", but the scenario is dash-38. 4ab288b says "No new Gherkin scenario was added", which a later commit reversed.
- The project squash-merges, so only the squash message matters.
- Confidence 75 · mechanical · Fix: write an accurate squash message.
Decisions for the author
- [standing] Per-tick samples vs. per-run summaries — non-blocking-improvement
- The ticket asks for recent serving runs. At 1 s snapshots, five consecutive non-zero ticks usually span about 5 s of a single burst.
- Grouping consecutive non-zero ticks from the same 240-snapshot history into runs (start age, duration, mean/peak tok/s, latency) would match the ticket more closely, at about the same cost.
- The current shape is not wrong, and both the code comment and the description say it is per-tick.
Positive signals
top_padcomes frompanel::padding_forrather than a re-derived threshold, so the reserved height cannot drift from whatbentodraws.has_observed_throughputkeeps the panel from claiming "idle" for new, expired or non-vLLM instances. Both placeholder branches are pinned: swapping the messages or droppingis_finiteeach fail a test.- Each layout guard (padding row,
BODY_FLOORcap,min_hcollapse) has a render test that fails when that guard alone is removed. format::ms_optreplaces two inline copies indraw_tablewith identical output (—).
Deployment notes
None
What this covered
- Read: all four changed files completely, against prw-base f6c8a8f…a87d1a8b. Also the surrounding code:
panel.rs(bento,padding_for, scrollbar),app/mod.rshistory handling,format.rsfreshness/gen_tps helpers,rocm-dash-coreobservation tracker, the demo generator, and the daemon tick cadence. - Fan-out: two independent read-only workers. One covered the code and unit tests; the other covered e2e plus the agent-instruction, history and code-comment passes. I ran the design pass, the CI read and Step 9 myself.
- Runs, all in the checkout's own
target/withRUSTC_WRAPPER=empty:- Clippy:
cargo clippy -p rocm-dash-tui --all-targets -- -D warningswas clean after touching the sources. - Tests: the 11 new tests plus 6 existing
detail_modal_*tests passed (17 run). - Mutations: 8 one-at-a-time mutations on the new production code. Every one except
.max(0)was caught by at least one test. - One render probe, which confirmed the e2e-step finding.
- Clippy:
- Per-test verdicts:
- With the production change reverted, the new unit tests do not compile, because they reference the new functions.
- Judged on behaviour alone, every new render test would fail on reverted code except
..._body_scrollbar_survives_..._short_terminal. That one guards the cap, not the feature. - The new e2e step fails on reverted code but passes on the idle placeholder; see Blocking.
- Sign-off: the
check_signoff.pytool exited 0, and all 4 commits carry sign-off. - Did not run:
- The dash-38 e2e scenario locally. CI's MI300X log shows it passing.
- The
feature_namingtest. Its rules were checked by hand: @id is unique, the dash-38 index does not collide. - Index-collision checks against any main newer than prw-base.
- The prior-changes pass, because earlier reviews are out of bounds by instruction.
- Not reconciled against the PR's discussion.
Earlier objections (from review 5630756475)
The findings above were reached blind. Each objection below was then settled by putting its wording and the author's reply to the reviewer after the review was finished. Those verdicts are anchored evidence, not findings.
- "no throughput data in the last 240s" was often untrue — settled. The placeholder now reads "no throughput data observed yet", with no window. The no-data and non-finite-only tests pin that branch. (The
!contains("240")assertion is not what guards it; see Non-blocking.) - Header-only panel on a short terminal — settled. The panel collapses to height 0 below
min_h(3 for the placeholder, 4 for header plus one row). A mutation settingmin_h = 0fails the short-terminal test. Not checked: no test drives the placeholder variant at exactly a 1–2 row cap; it is the same branch. - No Gherkin scenario (AGENTS.md §3) — holds in part. dash-38 now reaches the panel in a real dashboard and passed on the MI300X lane, but its step cannot tell rows from the idle placeholder (first Blocking finding). The two placeholder messages are covered by render tests that assert message and state together, not by a scenario; say so in the PR text, or add a scenario for them.
- TTFT/TPOT column swap and
is_finitemutations survived — settled. Swapping the cells failsdetail_modal_activity_row_shows_ttft_and_tpot_in_their_own_columns; looseningis_finitefails the non-finite-only history test.
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · a87d1a8
Review — needs work
Full review of the whole change (merge-base 63ecab2d…head a87d1a8b), via 2 independent fan-out passes (UI-code group: format.rs+instances.rs+panel.rs; e2e-test group: dash.feature+dash_steps.rs) plus manual verification of the layout arithmetic and CI status.
Implements part of ROCMAI-196 ("add a rolling summary of recent serving runs: tokens, tok/s, latency alongside the live gauge"). This change instead shows recent non-idle snapshot ticks (time-ago / tok/s / TTFT / TPOT), explicitly not discrete serving runs and with no token counts — the PR body discloses this scope narrowing itself, so it is not a claim mismatch.
Blocking
-
Required check
E2E tests (MI300X, current)is failing on this head commit — CI (gh pr view --json statusCheckRollup). Every other check (including the other four E2E hardware/OS lanes, clippy, coverage, commit-signatures) is green. I could not retrieve the job log to determine root cause:gh run view <run-id>andgh api .../actions/runs/<id>both return403 Forbiddenfor this self-hosted-runner job from here. Whether this is a flaky GPU-hardware lane or a real regression from this change is unresolved; it still needs to go green (or be explicitly waived with evidence it's unrelated) before merge.
Confidence 100 (CI's own report) · logic · Fix: inspect the MI300X job log directly (requires access this review didn't have) and either fix the regression or get the flake triaged/rerun. -
The new "recent throughput" panel silently drops rows (no "+N more" or similar) across a realistic band of terminal heights, reproducing the exact defect class this PR's own round-1/round-2 fixes targeted —
crates/rocm-dash-tui/src/ui/tabs/instances.rs:714-741(draw_detail'sactivity_h/BODY_FLOOR/min_harithmetic),crates/rocm-dash-tui/src/ui/modal.rs:38-50,94-99,109-114(centered_rect/centered_height/draw_popup_frame, which usespanel::popup's compact mode —Padding::ZERO, not the adaptivepadding_forthe comments ininstances.rsreason about).
Tracing the actual popup-sizing chain for a 5-row-full activity panel (content_rows=6, uncappedactivity_h=9) against terminal heightT:inner.height(detail popup's own content height) works out to roughlyfloor(0.85·T) - 2, and the cap at line 736 isactivity_h.min(inner.height - 8). ForT=20(a thoroughly ordinary size — e.g. a90x20or100x20pane) that givesinner.height=15, cappedactivity_h=7;panel::bento's ownpadding_forthen reserves 2 border rows + 1 top-padding row (since7>=5), leaving only 4 content rows for the needed 6 (header + 5 data rows) — 2 of the 5 activity rows (the oldest two) are silently clipped, with nothing on screen indicating they exist. AtT=17,activity_hcomputes to exactly4(themin_hfloor for a non-empty panel), which clears the collapse check but only has room for header + 1 data row — 4 of 5 rows dropped. Only outside roughlyT∈[17,22]does it either fully fit (T≳23) or fully collapse to 0 (T≲16, as the shipped90x16test exercises).
The two shipped regression tests bracket this band without landing in it:detail_modal_shows_all_activity_rows_up_to_the_cap(instances.rs:2616) uses160x48(well above the band, no capping at all), anddetail_modal_body_scrollbar_survives_a_full_activity_panel_on_a_short_terminal(instances.rs:2650,TestBackend::new(90, 16)) lands below it (full collapse to 0, not partial clipping) — and that test's own assertion (!out.contains("TOK/S") || out.contains("105"), line ~2692) would in fact accept a header rendered with only the newest row and none of the others, so even a test at the right size wouldn't currently catch partial clipping.
Confidence 85 (traced the complete arithmetic chain, including the previously-missingcentered_rect/draw_popup_framesizing code) · logic · Fix: either derive the actual available row count first and request only that many rows fromrecent_activity_rows(so the panel never claims to show more than it can), or add an overflow indicator (the crate already has this idiom —format::overflow_line, used elsewhere for "...and N more" — albeit for a different kind of truncation) so a user on a mid-size terminal isn't shown a shorter, unmarked list that looks like a sparse history rather than a clipped one. Add a regression test in theT∈[17,22]band either way.
Non-blocking
None.
Decisions for the author
- Height math for the activity panel is hand-derived in
draw_detailrather than owned by a shared helper — non-blocking-improvement.instances.rs:730builds a throwawayRect::new(0, 0, 1, raw_h)purely to borrowpanel::padding_for'stopfield, and the surrounding comment (instances.rs:726-729) is explicit that re-deriving this threshold instead of centralizing it is "exactly how the oldest activity row got clipped" the first time. A small helper onpanel.rs(e.g.reserved_rows_for(content_rows) -> u16) would make a second drift (like the Blocking finding above) harder to reintroduce by construction, rather than relying on a comment to prevent it.
Positive signals
- The idle ("has been idle", a genuine zero sample) vs. no-data ("no throughput data observed yet", never observed) placeholder split (
has_observed_throughputvs.recent_activity_rows,instances.rs:323-398) is a careful fix for a real false-claim risk, and every variant is pinned together with the state that produces it (e.g.detail_modal_shows_no_data_placeholder_when_history_has_only_non_finite_samples), exactly per this repo's own "message asserted together with the behavior" convention. - The commit history shows genuine engagement across two review rounds, each commit pairing one specific defect with one specific regression test (the padding/row-count fix, the
BODY_FLOORscrollbar-starvation fix, theformat::ms_optdedup) rather than a single undifferentiated "address feedback" squash.
Deployment notes
None.
What this covered
Read completely: crates/rocm-dash-tui/src/ui/format.rs, crates/rocm-dash-tui/src/ui/tabs/instances.rs (both full files, not just the diff hunks), crates/rocm-dash-tui/src/ui/panel.rs (unchanged, read as callee context), tests/e2e-cucumber/features/dash.feature, tests/e2e-cucumber/tests/e2e/dash_steps.rs — via two independent fan-out workers (one per file group above) plus my own direct reading — and crates/rocm-dash-tui/src/ui/modal.rs (pulled in manually to resolve the popup-sizing arithmetic one worker flagged as unavailable to it). Diff: 63ecab2d26ffb569ed857a0e7b0ab26a12bd5853...a87d1a8bb50edc4f2b967a6e7e5465e4c86c403d.
Passes run: grouped code passes (2, above), history (all 4 commit messages read and each one's claimed fix cross-checked against the diff), agent-instruction adherence (this repo's AGENTS.md read in full), code-comments (no outstanding TODO/FIXME in touched files), impact analysis (README.md, docs/ux-guidelines.md, coverage-floors.toml checked — no required doc update found; the README documents the dashboard only at the tab level, not per-panel), design review (one tradeoff surfaced above), and the ticket/description reconciliation (ROCMAI-196 retrieved directly via Jira).
Did not run: "prior changes" (no accessible record of review comments on earlier, unrelated PRs touching these exact files from this environment); cargo test/cargo clippy locally (no Rust toolchain available in this environment) — relied on CI's own report instead, which is green everywhere except the MI300X lane noted above. A candidate finding about the new e2e step's placeholder-exclusion predicate (dash_steps.rs:1634-1649 — format::gen_tps_detail_freshness can also emit "...s ago" text elsewhere on the same screen) was investigated in depth but traced out: the new scenario never injects the scrape-failure step that would produce that competing text, so it does not currently misfire; left out as below the reporting threshold.
Not reconciled against this PR's own review-comment discussion (deliberately not read, per method); existing reviews on this PR were checked only to confirm this round is not a duplicate of one already posted for this exact head commit.
|
Addressed review 5664058417 in 889e1c2. Blocking
Non-blocking, fixed
Left as is (deliberate)
Checks run locally: |
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 889e1c2
Review — no blocking findings
Read the full diff between the merge-base and the current head: the new "recent throughput" panel and its supporting helpers in crates/rocm-dash-tui/src/ui/tabs/instances.rs, the shared format::ms_opt helper, and the new/updated e2e scenario and step. Also read the surrounding code it depends on (panel::bento/padding_for, the Lemonade collector's container-id scheme, the app's history/merge logic) to check the trickier height-budget and history-matching logic rather than just the diff in isolation. Overall this is a solid, well-tested addition — I found no issue that should block merge, just a few things worth a look.
Blocking
None
Non-blocking
- "Recent throughput" can show a different model's stale samples after a Lemonade hot-swap.
recent_activity_rows/has_observed_throughput(crates/rocm-dash-tui/src/ui/tabs/instances.rs:344and:391) match purely oncontainer_idagainst the full retained history. The Lemonade collector's id is derived only from host+port (crates/rocm-dash-collectors/src/lemonade.rs:114,lemonade_container_id), so it stays the same across a model swap on the same endpoint, and nothing clears or filters history when the loaded model changes. Right after a swap, the panel can briefly surface a burst from the previous model under the newly-loaded model's detail view. A low-cost mitigation would be to also require the matching snapshot'smodel_nameto equal the currently-displayed instance's model_name, not justcontainer_id. - Three near-identical "scan history by container_id" helpers, with inconsistent iteration order.
recent_activity_rows(:344),has_observed_throughput(:391) and the pre-existingbuild_kv_heatmap_rows(:297) all walkhistorylooking up entries bycontainer_id, but the first scans newest-first via.rev()while the third scans oldest-first. Worth consolidating into one helper with a documented iteration-order contract so the next addition doesn't have to re-derive it (or quietly pick the wrong order). - Height-budget cap hardcodes sibling layout sizes as literal numbers. The short-terminal guard at
:749(inner.height.saturating_sub(4 + 1 + BODY_FLOOR)) repeats the4and1used by theConstraint::Length(4)/Constraint::Length(1)chunks defined a few lines later (:758,:761). If that layout is ever resized, nothing ties the two together, soBODY_FLOOR's guarantee (the body panel keeps enough room for its scrollbar) could silently stop holding.
Decisions for the author
- Per-tick samples vs. grouped serving runs — tradeoff, already called out in the PR description. Per-tick is simpler and already reviewed; grouping into runs (start/duration/peak/avg) would read closer to "serving runs" but is a bigger change. Reasonable to leave as a possible follow-up.
- Vertical cost of the new panel — tradeoff. At its full 5-row cap the panel can reserve up to ~9 rows (2 border + top padding + header + 5 rows) before the
BODY_FLOORcap engages, which is real real-estate on a small terminal. The existing cap/collapse logic handles the extreme case reasonably; worth confirming the common case (a populated panel on a modest terminal) still looks good. - No elision indicator when the cap trims rows on a short terminal — non-blocking improvement. Once
BODY_FLOORshrinks the panel, it can end up showing fewer than the 5 captured rows with no "+N more" hint. Not wrong (it degrades gracefully rather than breaking), but a small UX polish opportunity.
Positive signals
- The idle-vs-no-data split is tested precisely at the state that drives the choice (
detail_modal_shows_no_data_placeholder_when_history_has_no_throughput_sample/..._shows_idle_placeholder_when_history_has_only_zero_samples), rather than just pinning the wording. - The padding-drift regression this PR itself fixes (oldest row getting clipped once the box crosses the 5-row padding threshold) has a dedicated regression test with a comment explaining exactly what it guards against.
- TTFT/TPOT formatting was pulled into a shared
format::ms_optand reused by both the existing table and the new panel instead of duplicating the sameis_finite/format logic a third time.
Deployment notes
- CI on the current head is green on everything that has reported so far (clippy, build/tests on Linux and Windows, coverage, docs build, commit-signature check); the GPU-hardware E2E lanes were still in progress/pending at review time with nothing failing.
- Purely additive UI behavior behind the existing instance-detail modal; no config, schema, or protocol changes, so no migration or compatibility concerns.
- The Lemonade staleness point above only applies to deployments using the Lemonade engine with model hot-swapping on a single endpoint; it doesn't affect vLLM/container-backed instances, which get distinct ids per container lifecycle.
What this covered
Read in full: crates/rocm-dash-tui/src/ui/format.rs, crates/rocm-dash-tui/src/ui/tabs/instances.rs (diff plus the whole file for context), tests/e2e-cucumber/features/dash.feature, and tests/e2e-cucumber/tests/e2e/dash_steps.rs, diffed between the merge-base and the current PR head. Also read the supporting code the diff depends on (ui/panel.rs's padding_for/bento, the Lemonade collector's id scheme, and the app's history/merge logic) and the history of the touched files.
Did not run: no Rust toolchain or network access available in this environment, so nothing was built, linted, or executed — all conclusions above come from reading and manually tracing the code, not from running cargo test/clippy/the e2e suite myself. CI's own results (above) are taken from the PR's reported check statuses.
Not reconciled against the existing review thread on this PR beyond reading it for scope; the findings above were derived independently from the code itself.
Addresses rominf's review (5625116225) on PR #721: - Fix activity_h to account for panel::bento's top-padding row once the box reaches 5 rows tall, so RECENT_ACTIVITY_CAP = 5 actually shows all 5 rows instead of clipping the oldest one. - Split the empty-history placeholder into "has been idle" (a real Some(0.0) sample was observed) versus "no throughput data" (never observed, past the observation's validity window, a managed non-vLLM engine, or a freshly-seeked replay) so the message can't claim idle when nothing is actually known. - Render recent-activity tok/s through format::gen_tps_cell so held (carried-forward) samples get the same held marker as every other gen_tps surface in this file. - Add a short-terminal regression test for the BODY_FLOOR cap itself (overflowing launch_args + a full activity panel), proving the scrollbar thumb survives. - Cover a non-finite (NaN/Inf) tick in the row-selection test. - Add format::ms_opt to de-duplicate the TTFT/TPOT formatting that was copy-pasted between draw_instances_table and render_recent_activity. - Rename the panel "recent activity" -> "recent throughput" so it isn't mistaken for the Home tab's unrelated "Activity * node" panel. No new Gherkin scenario was added for the panel: demo mode's gen_tps oscillates continuously and is always positive with gen_tps_observation: None, so the PTY-driven E2E framework can't deterministically exercise the idle-vs-no-data distinction without substantial new step-definition plumbing. Existing unit/render tests already cover both the selection rules and both placeholder messages. Signed-off-by: Juho Vainio <juho.vainio@amd.com>
889e1c2 to
710d2bc
Compare
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 710d2bc
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Review — needs work
Full review of the whole change (base db331bd0, head 710d2bcc).
Implements part of ROCMAI-196 ("Add a rolling summary of recent serving runs (tokens, tok/s, latency)"): this shows recent non-idle snapshot ticks instead of discrete serving runs and omits token counts, which the PR description discloses explicitly as a deliberate scope choice (no per-request telemetry exists in the collector pipeline).
Blocking
-
PR description names the wrong scenario for the new e2e coverage —
tests/e2e-cucumber/features/dash.feature:678
The description says "New e2e scenario dash-38," butdash-38("A user reads the warning behind the header badge") already existed before this PR and is unrelated. The scenario this PR actually adds isdash-39 - Instance detail lists recent throughput samples for a serving model. A reader following the description to find the new coverage lands on the wrong, pre-existing scenario.
Confidence 100 · mechanical · Fix: correct the description to referencedash-39. -
Live CI is not fully green at this commit — self-hosted E2E lanes
For head710d2bccd7a7c9fd7060049a3ea6033dc4c4db01, workflow run 40912189057 ("E2E self-hosted") showsE2E tests (Strix Halo, Ubuntu, release, current)andE2E tests (Strix Halo, Ubuntu, nightly, current)with conclusionCANCELLED, andE2E tests (W7900, current)stillQUEUED(never completed) as of this review. The default CI pipeline (build, clippy, fmt, unit tests, coverage) and theE2E tests (MI300X, current)lane — the Linux lane the PR's own test plan names as needed for the new@requires-os:linuxscenario — are green. Still, three lanes for this exact head commit have not reported a pass.
Confidence 100 · mechanical · Fix: rerun or otherwise resolve the cancelled/queued self-hosted lanes before merge.
Non-blocking
None
Decisions for the author
- "Recent non-idle snapshot ticks" vs. the ticket's "recent serving runs (tokens, tok/s, latency)" — tradeoff
The implementation surfaces per-tick throughput/TTFT/TPOT samples rather than discrete request runs, and shows no token counts, because the collector pipeline has no per-request telemetry. This is a reasonable, clearly-disclosed stop-gap that ships real signal now; the alternative (waiting for collector-level per-request telemetry) is larger and blocks the ticket indefinitely. Confirm this partial scope is acceptable as a first step, with the fuller per-run summary tracked separately if the telemetry work lands later.
Positive signals
instances.rs'sactivity_hcomputation reusespanel::padding_forinstead of re-deriving itsheight >= 5threshold, and the reservation is self-consistent: hand-tracingcontent_rowsfrom 1 through 6 (0 through 5 activity rows) shows the top-padding crossing point always lands on the same side for the pre-cap and post-reservation heights, so the previously-possible "oldest row clipped" failure mode is actually closed, not just asserted to be — including by thedetail_modal_shows_all_activity_rows_up_to_the_capregression test.has_observed_throughputcorrectly separates "never observed" (None/non-finite only) from "genuinely observedSome(0.0)" so an idle instance isn't confused with one lacking telemetry altogether, backed by distinct tests for each of the no-data / idle / non-finite-only / active / held / cap-boundary states.format::ms_optis a clean, behavior-preserving extraction of the previously-duplicated TTFT/TPOT formatting (confirmed the old inline fallback and the new helper both use the same"—"em-dash), now shared betweendraw_tableand the new panel.
Deployment notes
None beyond the CI gap noted above.
What this covered
Read crates/rocm-dash-tui/src/ui/tabs/instances.rs and crates/rocm-dash-tui/src/ui/format.rs in full at pr-721, plus crates/rocm-dash-tui/src/ui/panel.rs in full for the bento/padding_for/render_box cross-check; tests/e2e-cucumber/features/dash.feature and tests/e2e-cucumber/tests/e2e/dash_steps.rs in full; the PR's 5-commit history (0c8c2cc3…710d2bcc) and repo-wide history for panel.rs/instances.rs/format.rs; AGENTS.md; live CI via GitHub's API; and ticket ROCMAI-196. Leak scan of the diff was clean.
Did not run: no local cargo/rustc toolchain in this pass — code-logic findings were verified by direct reading and hand-tracing against the actual source rather than by executing the test suite. Not reconciled against this PR's own prior review threads.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · c191190
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change (diff, scoped by file: crates/rocm-dash-tui/src/ui/format.rs, crates/rocm-dash-tui/src/ui/instance_detail.rs, crates/rocm-dash-tui/src/ui/tabs/instances.rs, tests/e2e-cucumber/features/dash.feature, tests/e2e-cucumber/tests/e2e/dash_steps.rs).
Implements part of ROCMAI-196 ("Add a rolling summary of recent serving runs (tokens, tok/s, latency) alongside the live gauge"): shows recent non-idle snapshot ticks rather than discrete serving runs, and shows no token counts. The PR description discloses this scope limitation itself, consistent with the ticket's broader ask.
Blocking
-
The "recent throughput" panel silently drops its oldest rows on an ordinary terminal size, below the
RECENT_ACTIVITY_CAPit claims to show —crates/rocm-dash-tui/src/ui/instance_detail.rs(draw_detail'sactivity_hcapping, ~lines 69-93;render_recent_activity, ~lines 211-233)
TheBODY_FLOORcap (activity_h.min(inner.height.saturating_sub(4 + 1 + BODY_FLOOR))) reduces the panel's height for the body's sake, butrender_recent_activitystill receives and attempts to render the uncapped row list (up to 5) — it is never re-sliced to what the capped height can actually hold. Tracing the real geometry fromui/mod.rs'sbody = outer[1](terminal height − 4) throughcentered_rect/centered_height/scale_pctandpanel::padding_for: for a body height in roughly 17–22 rows (i.e. ordinary terminal heights around 21–26, including the ubiquitous 80×24 default), the cap engages but does not collapse the panel (it stays ≥ the 4-row minimum), so it renders — just with 1–3 of the 5 rows silently cut off, no truncation indicator. Neither existing render test exercises this band: the "all rows up to the cap" test uses a spacious 160×48 (cap never engages), and the short-terminal test uses 90×16 (the panel fully collapses to 0 well before this partial-clip band). This is the same clipping failure class this PR's own history (commitff43669) already found and fixed once, reintroduced by a path the fix didn't cover.
Confidence 85 · logic · Fix: re-slicerowsto what the final (post-cap) height can hold before rendering, or derive the cap from how many rows actually fit rather than capping the box height independently of the row count; add a render test at a body height in the 17–22 row band with 5 active ticks. -
No test exercises the empty-activity + short-terminal collapse branch —
crates/rocm-dash-tui/src/ui/instance_detail.rs:92-93(let min_h = if activity.is_empty() { 3 } else { 4 };)
This repo's own AGENTS.md requires a test per lifecycle state for a new placeholder/state display. The non-empty short-terminal collapse (min_h = 4) is covered bydetail_modal_body_scrollbar_survives_a_full_activity_panel_on_a_short_terminal(90×16), but every placeholder/no-data/idle test in this file uses a spacious 160×48 terminal — none combines an empty activity list with a short terminal, so themin_h = 3branch is never exercised by any test.
Confidence 85 · architectural/test-coverage · Fix: add a render test with empty history on a short terminal (e.g. 90×16) asserting the placeholder is shown or the panel cleanly collapses. -
PR description's test plan cites the wrong scenario ID — PR description ("New e2e scenario dash-38 …")
The actually-added scenario isdash-39(tests/e2e-cucumber/features/dash.feature:678);dash-38is a pre-existing, unrelated scenario ("A user reads the warning behind the header badge",dash.feature:660). A reviewer or future maintainer checking "is dash-38 the new coverage" finds unrelated existing content instead.
Confidence 100 · mechanical · Fix: correct the description todash-39.
Non-blocking
- Stale module-doc panel enumeration in three places —
crates/rocm-dash-tui/src/ui/instance_detail.rs:5-6(its own//!header still lists only "a summary strip, scrollable launch-args and env-vars panes, and a footer"),crates/rocm-dash-tui/src/ui/tabs/instances.rs:5-7(lists "model / partition / launch_args / env / log" for the detail modal), anddocs/architecture.md'sdraw_detailpanel inventory — none mentions the new "recent throughput" panel, even though the inline layout comment insidedraw_detailitself was updated. Confidence 85 · Fix: add the new panel to all three enumerations. - Gherkin comment names the wrong file for its own render tests —
tests/e2e-cucumber/features/dash.feature:675-676says the placeholder wording and short-terminal collapse "are pinned by render tests inui/tabs/instances.rs"; those tests actually live inui/instance_detail.rs'smod tests(confirmed directly —instances.rsno longer contains the detail-modal test module since the file split). Confidence 100 · Fix: correct the path.
Decisions for the author
- Pure arithmetic vs. full-render testing for the activity panel's sizing — non-blocking-improvement. The
raw_h/top_pad/activity_h/BODY_FLOORcalculation is intricate enough that it has already needed three fix rounds within this PR, and the blocking finding above slipped through a fourth time. Extracting it into a small pure function with unit tests across the full (row-count × terminal-height) input space would catch this class directly, rather than relying on full-render snapshot tests at a few hand-picked sizes. - Snapshot-tick granularity vs. discrete serving-run telemetry — tradeoff, already disclosed by the author in the description ("there's no per-request telemetry in the collector pipeline... happy to adjust the granularity if a different shape is wanted"). Flagging only to confirm this is the intended shape for this iteration of ROCMAI-196 given the ticket literally asks for tokens/tok-s/latency per "serving run."
Positive signals
format::ms_optis a clean, verified behavior-preserving extraction — confirmed it reproduces the exact"—"dash and{v:.0}msformatting the inlined code ininstances.rs::draw_tableused before.- Held-tick de-duplication correctly ages a collapsed row from
observed_atrather than the re-publishing tick, with a dedicated, well-targeted regression test. - The "no data yet" vs. "has been idle" placeholder split is a thoughtful, well-tested piece of UX honesty — it avoids falsely claiming an instance has been idle when it simply has no observed sample yet.
- The new e2e step's row-detection predicate robustly disambiguates a real data row from the panel title, the placeholder text, and the summary's own "held · Ns ago" freshness line.
Deployment notes
- CI on the reviewed commit is green across lint/build/unit tests/clippy/coverage, but several hardware E2E lanes (including MI300X, which the PR description names as the lane needed to exercise the new scripted-metrics scenario) were still queued at review time rather than completed — confirm they go green before treating the new scenario as proven.
What this covered
Read the full diff between the merge-base e341aebe78b5c4e61d7d1098a7f14b8a15adb8ca and the PR head c191190e32e5a9cf385295e8669b7a70c9ac7473 (a later commit merged in an unrelated main-branch refactor that physically relocated this code between files; the diff command used nets that out to the true 5-file, 585/16-line scope). Read all 5 changed files completely at the PR head, plus full reads of panel.rs (padding_for, bento/render_box, centered_rect/centered_height/scale_pct, popup) and the relevant parts of metrics.rs and ui/mod.rs's call site, tracing the layout arithmetic end-to-end from the real terminal-size entry point. Ran: agent-instruction adherence (root AGENTS.md), history (git log on the touched files and their pre-split home in tabs/instances.rs), code-comment accuracy, and impact analysis/beyond-diff completeness (grep for other TTFT/TPOT formatting sites, other consumers of the new identifiers, doc and characterization-test staleness).
Did not run cargo test/cargo clippy directly (no local build); relied on the PR's own CI, which is green for the reviewed commit except several still-queued hardware E2E lanes noted above. Read the PR description and Jira ticket ROCMAI-196 (via the Jira MCP tool) at Step 9, after the findings above were reached blind.
Not reconciled against any PR discussion, review comments, or prior review rounds — this review was blind to all of that throughout, per method.
🤖 Generated with Claude Code
|
Addressed review 5740842110 in 5e4d1e2:
Left as is: snapshot-tick granularity vs. per-request telemetry — still the disclosed scope for this iteration, since the collector has no per-request data. Verification: |
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · bc90ec6
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change.
Implements part of ROCMAI-196 ("add a rolling summary of recent serving runs") by scope — the PR's own description explains why it shows recent non-idle snapshot ticks rather than discrete serving runs with token counts (no per-request telemetry in the collector pipeline); that narrowing is stated, not hidden, so it is not a finding.
Blocking
-
CI is red: a new scenario duplicates an existing scenario's naming index, failing a committed drift-guard test on two lanes —
tests/e2e-cucumber/features/dash.feature:702(new scenario, titleddash-39 - Instance detail lists recent throughput samples for a serving model) collides with the pre-existingtests/e2e-cucumber/features/dash.feature:684(dash-39 - A user reads the warning behind the header badge).tests/e2e-cucumber/tests/feature_naming.rsenforces that scenario names are sequential per feature file and globally unique by index; the new scenario is declaration-position 40 but is nameddash-39 -instead ofdash-40 -. Confirmed two ways: (1) hand-reproducing the file's own parsing/assertion logic against the PR head content reproduces the exact panic text, and (2) GitHub Actions already shows it failing for real —Test (affected crates)(Linux) andWindows build and test (debug)both fail withthread 'scenario_indexes_are_unique_across_the_suite' panicked ... index "dash-39" is used by both dash.feature and dash.featureandthread 'scenario_names_are_indexed_sequentially_per_feature' panicked ... dash.feature: scenario 40 is named "dash-39 - ..." but must start with "dash-40 - ".
Confidence 100 · mechanical · Fix: rename the new scenario's title to start withdash-40 -. -
New "recent throughput" rows show a fabricated "0s ago" instead of a real age when the dashboard is disconnected —
crates/rocm-dash-tui/src/ui/instance_detail.rs:254(let age_s = snap_ts.map_or(0, |ts| (ts - r.ts).num_seconds().max(0));thenformat!("{age_s}s ago")), fed bysnap_tscomputed at:68(state.latest.as_ref().map(|s| s.timestamp)). OnClientMsg::Disconnected(crates/rocm-dash-tui/src/app/event_loop.rs:445-448), onlystate.latestis cleared —state.instancesandstate.historyare left populated with stale data — sodraw_detail's early-return guard (state.instances.is_empty()) does not trigger, and the panel keeps rendering old history rows withsnap_ts == None, defaulting every row's age to literally0s agoregardless of how stale the data actually is. This is reachable in production and no test exercisesrender_recent_activity/draw_detailwithsnap_ts: Noneand a non-empty activity list. It also contradicts this same file's own established, safer precedent for the identical condition:render_summary's freshness line (:297,format::gen_tps_detail_freshness(inst.gen_tps_observation.as_ref(), snap_ts)) explicitly returns"held"for a missing snapshot timestamp instead of fabricating a duration.
Confidence 85 · logic · Fix: render a safe placeholder (e.g. reuse the "held"-style treatment, or"—") whensnap_tsisNone, instead of defaulting the subtraction to0.
Non-blocking
None
Decisions for the author
- "Recent throughput" has no time-window cutoff, only a 5-row count cap — tradeoff.
recent_activity_rows/has_observed_throughput(instance_detail.rs:150-206) scan the full 240-tick in-memory history with onlyRECENT_ACTIVITY_CAPas a limit, so a long-idle instance can keep showing an old burst under a "recent" heading for as long as that entry survives the ring buffer, rather than falling back to the "has been idle" placeholder. The displayed per-row age does let a user self-judge staleness, so this may be intentional; confirm it is. - A Held tick's displayed age and its TTFT/TPOT values can come from different ticks — tradeoff. In
recent_activity_rows(instance_detail.rs:150-191), the dedup keeps the newest tick in a Held streak (and itsttft_ms/tpot_ms), while the row's displayed age is aged from the originalgen_tps_observation.observed_at— correct for what the comment claims aboutgen_tpsspecifically, butttft_ms/tpot_mscarry no such freshness provenance of their own, so a row's age label and its latency columns are not guaranteed to describe the same instant. Confirm this is the intended semantic (freshest-available latency, paired with the gen_tps-specific age) rather than an oversight. - Full-history scan on every render frame — non-blocking-improvement.
recent_activity_rowsandhas_observed_throughputeach independently walk up to 240 history entries (with a nested per-snapshot instance lookup) every time the Detail modal redraws. Not incorrect, but could be computed once per snapshot arrival and cached onAppStateif this becomes a hot path.
Positive signals
- The new panel reuses
panel::bentorather than hand-rolling aBlock, and both placeholder messages are computed directly from the samehas_observed_throughputboolean the branch itself uses, so the message cannot drift from the underlying decision — both satisfy this repo's own dash-TUI guardrails. format::ms_opt's extraction fromtabs/instances.rs's inline TTFT/TPOT formatting was verified byte-for-byte behavior-preserving (same em-dash glyph, same row-sibling formatting), a clean, low-risk simplification.- The panel's non-trivial layout budget math is backed by specifically-named regression tests pinned to past bugs (padding-row-at-5-rows,
BODY_FLOORstarvation, short-terminal collapse), each with a comment explaining what it guards against.
Deployment notes
CI is currently red on two separate lanes (Linux "Test (affected crates)" and "Windows build and test (debug)") for the same root cause (see Blocking #1); this must not merge until both are green. No migration or breaking API/behavior change otherwise.
What this covered
Every file in the change, completely: crates/rocm-dash-tui/src/ui/format.rs, crates/rocm-dash-tui/src/ui/instance_detail.rs, crates/rocm-dash-tui/src/ui/tabs/instances.rs, docs/architecture.md, tests/e2e-cucumber/features/dash.feature, tests/e2e-cucumber/tests/e2e/dash_steps.rs — plus surrounding/caller code (app/event_loop.rs, app/mod.rs, rocm-dash-core::metrics, tests/e2e-cucumber/tests/feature_naming.rs) needed to judge the changes in place. Base e6d2f7c0d7b9a8051a475dfaca84756e5b97aecf … head bc90ec6da6973b45ec0e4941967091582038d87c.
Fan-out: two code-coupling passes ran as independent workers (format.rs+instance_detail.rs; tabs/instances.rs+docs/architecture.md), as did one combined source-reading pass (agent-instruction adherence against root AGENTS.md, git history, prior-changes, code-comments) across all six files. A fourth worker (the e2e test-quality pass over dash.feature+dash_steps.rs) could not be spawned after four attempts — an environment-wide concurrent-subagent cap was saturated by other sessions sharing this sandbox throughout, unrelated to this review's own work — so that pass was run directly in the orchestrating context instead of as an independent worker, per the method's fallback rule; it is the source of the dash-39 finding, independently corroborated by the source-reading worker and by live CI.
Disclosure on blindness: existing GitHub reviews on this PR were listed once, mechanically (login/state/commit-id/date only, via a script that never printed review body text), solely to confirm the PR had moved since the last review and to avoid double-posting — no review body text was read before findings were settled. Fetching PR metadata for scope resolution incidentally returned the PR description earlier than Step 9 formally calls for; it was not used to shape any finding and was (re-)examined and reconciled only at Step 9, after all findings above were already settled from the code and from live CI.
Did not run cargo test/cargo clippy locally (the PR ref was never checked out, per method rules); the two blocking findings instead rest on direct code/control-flow tracing plus the PR's own live CI results (get_check_runs, taken at face value). Not reconciled against any PR discussion/thread text; this report stands independent of it.
🤖 Generated with Claude Code
jussielo-amd
left a comment
There was a problem hiding this comment.
Thanks @juhovainio, the placeholders, the bento-padding arithmetic and the activity_layout sweep test are nicely done. A few blockers:
Blocking
- Scenario id
dash-39is already taken, and CI is red because of it.dash.feature:684isdash-39 - A user reads the warning behind the header badge; the new scenario at:702is alsodash-39.feature_naming.rsfails (scenario_indexes_are_unique_across_the_suiteand the sequential-name check), which is whyTest (affected crates)andWindows build and test (debug)are red.mainis already atdash-41, so after rebasing it should bedash-42; the PR text and test plan need the new number too. The PR is alsodirtyagainstmain(it touches filesmainhas since changed). - The held-tick dedup keeps the wrong tick. In
recent_activity_rows(instance_detail.rs~l.179) iteration is newest-first and.scankeeps the first row of eachobserved_atgroup, i.e. the later Held tick. As I read the cadence (a snapshot per 1s tick, a scrape per 2s), a steadily serving instance alternates Fresh(T) and Held(observed_at=T), so most rows would render as86*even though each scrape was fresh, and on a failed scrape the Held tick has no TTFT/TPOT (—). The dedup test only assertsts, so it stays green. Keep the Fresh (or oldest) tick of each group and assert its freshness and latency. detail_modal_body_scrollbar_survives_a_full_activity_panel_on_a_short_terminalpasses with the panel removed. At 90x16activity_layoutreturns(0,0)(inner height 11), so the panel collapses; the comments about a "full 5-row panel" don't match, and!out.contains("TOK/S") || out.contains("105")is trivially true. Please retarget it to a height where the panel is drawn at the floor (inner 13 →(4,1)) and assert on the panel. Also: dropping thecontainer_idfilter inrecent_activity_rows/has_observed_throughputstays green since each snapshot holds one instance; a sibling-instance fixture would pin it.
Non-blocking (UX)
- At 80x24 the panel takes the space of the scrollable body: it had ~10 content rows before and has 2 with five active rows. Capping at ~3 rows on small terminals would keep launch args/env visible.
newest k of nreports the already-capped count (take(5)), so "newest 3 of 5" can mean 40 samples exist. Drop "of n" or count before capping.- The held
*in this panel has no legend, unlike the table, Home and Observe (HELD_LEGEND). Fixing #2 makes it rarer, but the legend would still help. - The panel disappears silently when there is no room for it.
- Per AGENTS §3, the idle placeholder is reachable end to end with
MetricsMode::RunningIdle; the scenario comment suggests it covers the idle/no-data split, but it only asserts a populated row. Either add a second scenario or reword the comment.
🤖 by agenthub on AMD AgentHub
|
Addressed review 5781305641 (and the open blocking items from 5754767620) in 3488640, which also merges the current Blocking
Non-blocking
Left as is
Checks: |
jussielo-amd
left a comment
There was a problem hiding this comment.
Thanks @juhovainio, this has come a long way. I checked the points from my last round against this head and they are resolved: the scenario is now dash-42 and feature_naming is green, the Held-tick dedup keeps the scrape's own Fresh tick, and the scrollbar test now targets 90x18 and the sibling-instance filter is pinned. CI is green so far (only the hardware lanes are still pending). One blocker is left, and it is a UX one.
Blocking
- The panel starves the launch args / env panes at common terminal sizes (inline on
BODY_FLOOR).
Non-blocking
- Held
*has no legend in the panel, and the "no throughput data observed yet" wording can mislead (inline). - The PR description is out of sync: the scenario is
dash-42(notdash-39), it carries only@requires-os:linux(no MI300X tag), and the listed test filtertabs::instances ui::formatdoesn't select the newui::instance_detailtests. - Small test nits: in the 90x18 and 160x48 tests the timestamps count backwards (
SNAP_SECS - 10 - i), so105is actually the oldest row rather than the newest; and the.max(0)age clamp has no test.
🤖 by agenthub on AMD AgentHub
|
|
||
| /// Rows the detail body needs reserved below it (border pair + a content row) | ||
| /// so its scrollbar can still render when the recent-activity panel is capped. | ||
| const BODY_FLOOR: u16 = 3; |
There was a problem hiding this comment.
Blocker (UX): BODY_FLOOR = 3 lets the panel take everything above one content row of the body, so it crowds out the launch args / env panes, which are the modal's main content.
Tracing centered_rect(85,85,120,36) + activity_layout with 5 active samples:
- 80x24: panel is ~7 rows, the body keeps ~1-2 visible lines (about 7 before this PR)
- 80x30: ~3 lines (about 12 before)
It still scrolls, but reading launch args at the default terminal size is effectively broken, and the only cue is the ↑/↓ scroll hint. The 90x18 test accepts a 1-row body.
Suggest a larger floor (e.g. >= 6 rows or half the inner height), or collapsing to the 1-row placeholder/summary below a height threshold, plus a test at 80x24 asserting several launch-arg lines stay visible.
| || "—".to_string(), | ||
| |ts| format!("{}s ago", (ts - r.ts).num_seconds().max(0)), | ||
| ); | ||
| let tps = format::gen_tps_cell(r.gen_tps, r.gen_tps_observation.as_ref()); |
There was a problem hiding this comment.
Non-blocking (UX): a held row prints 86*, but nothing in the panel explains the *. The other surfaces (instances table, Home, Observe, services manager) show format::HELD_LEGEND when a visible cell is held, and format.rs asks every held-marker surface to do so. Maybe render it when any shown row is held (and reserve that row in activity_layout).
| let msg = if has_observed_throughput { | ||
| "no recent throughput — instance has been idle".to_string() | ||
| } else { | ||
| "no throughput data observed yet".to_string() |
There was a problem hiding this comment.
Non-blocking (UX): "observed yet" suggests data is on its way, but for engines that never get gen_tps wired up (managed non-vLLM) it stays that way forever. It can also sit right under a live gen: 100 tok/s in the summary when an instance was just discovered and history is still empty. Something neutral like "no throughput samples" avoids both.
Add a "recent throughput" panel to the instance detail modal that lists the last few snapshot ticks where the instance had non-zero generation throughput, with tok/s and TTFT/TPOT at that time (ROCMAI-196). The live gen gauge mostly reads 0 tok/s between bursts, with nothing showing what the last few serving runs looked like. Built from the already-collected rolling snapshot history: idle ticks are filtered out, held ticks collapse to the scrape's own tick, and the newest five are kept. The panel is sized together with its row count and leaves the launch args / env body at least max(6, half the modal) rows, collapsing on shorter terminals. A `*` held row carries the shared held legend in the panel title. Covered by scenario dash-42 and render tests in ui/instance_detail.rs. Signed-off-by: juhovainio <juho.vainio@amd.com>
3488640 to
44b9e2d
Compare
|
Addressed review 5783507958 in 44b9e2d (history is now a single commit rebased onto current Blocking
Non-blocking
|
jussielo-amd
left a comment
There was a problem hiding this comment.
Thanks @juhovainio, nice follow-up. I checked each point from my last round against this head and all are resolved: the body floor now keeps the launch args / env panes readable (the layout no longer starves them at the sizes I traced), the held * legend is shown in the panel title, the placeholder wording is neutral, the test fixtures run oldest to newest, the .max(0) clamp has a test, and the PR text now matches the scenario id and tags. CI is green on this head, and I found no blocking issues.
Non-blocking suggestions are inline (held TTFT/TPOT marker and reuse of latency_cell, test geometry vs. the real 80x24 body, silent collapse at 80x24, legend dropped on narrow terminals, the "idle" wording). Small extras:
- The PR text says "at 80x24 the panel shows one row"; in the real app (body is 20 rows) it is hidden when there is data, so that sentence could be adjusted.
detail_modal_empty_history_on_short_terminal_...asserts the absence of "no throughput data", but the placeholder is now "no throughput samples", so that half of the assertion can't fail.body_floor's.max(6)has no effect (below 12 rows the panel already collapses), so the doc line "keeps at least 6 rows" isn't strictly true.
🤖 by agenthub on AMD AgentHub
| |ts| format!("{}s ago", (ts - r.ts).num_seconds().max(0)), | ||
| ); | ||
| let tps = format::gen_tps_cell(r.gen_tps, r.gen_tps_observation.as_ref()); | ||
| let ttft = format::ms_opt(r.ttft_ms); |
There was a problem hiding this comment.
Non-blocking (UX/consistency): TTFT/TPOT come from their own trackers, so a row with a fresh tok/s can carry a held latency from an earlier scrape, and here it is printed with no * (and the title legend only looks at the tok/s observation). The Instances table handles this with format::latency_cell(ms, observation), which marks held values and feeds the legend. Carrying ttft_observation/tpot_observation in ActivityRow and using latency_cell would fix it, and then ms_opt (which is the observation-less subset of latency_cell) can go. The test fixture for the Held tick ("carries no latencies of its own") is also something the runner doesn't actually produce.
| .unwrap(); | ||
| state.history.push_back(snap); | ||
| } | ||
| let mut term = Terminal::new(TestBackend::new(80, 24)).unwrap(); |
There was a problem hiding this comment.
Non-blocking (test fidelity): draw_detail is called with the app's body rect (terminal minus 3 header rows and 1 footer row), but this test, and the 90x18 / 100x28 ones, pass f.area(), so each is 4 rows taller than the terminal size in its name. At a real 80x24 the body is 20 rows, inner is 15, avail is 3, and a data panel (min 4 rows) collapses, so contains("recent throughput") would not hold there. Driving these through ui::draw, or passing Rect of height rows - 4, would make them match what users see. The PR text ("at 80x24 the panel shows one row") has the same off-by-4.
| /// handed rows it can't draw. A panel too short for its header plus one row | ||
| /// (or the placeholder line) collapses to `(0, 0)` so the body gets the rows. | ||
| fn activity_layout(n_rows: usize, inner_h: u16) -> (u16, usize) { | ||
| let avail = inner_h.saturating_sub(4 + 1 + body_floor(inner_h)); |
There was a problem hiding this comment.
Non-blocking (UX): the body floor fixes the starvation, thanks. One side effect at the real 80x24 default: a panel with data needs 4 rows and avail is 3, so it disappears silently, while the 3-row empty placeholder still fits and takes 3 of the 7 launch-arg rows (7 -> 4, so env vars start to need scrolling). The panel is hidden where it has the most to say and shown where it has nothing. Collapsing the placeholder at the same threshold as the data panel (or a one-line hint like "recent throughput hidden - enlarge terminal") would keep that consistent.
| .is_some_and(|m| m.freshness == rocm_dash_core::metrics::ObservationFreshness::Held) | ||
| }) { | ||
| title.push_str(" · "); | ||
| title.push_str(format::HELD_LEGEND); |
There was a problem hiding this comment.
Non-blocking (UX): panel::bento drops a title that doesn't fit (title_fits), and the longest form here ("recent throughput (newest N) · * = held (prior scrape window)") is ~61 chars, so on terminals narrower than about 80 columns the label, "(newest N)" and the legend all vanish together, leaving * rows unexplained. Picking a shorter variant (e.g. * held) when it doesn't fit, or putting the legend in the body, would avoid that.
| if inner.height == 0 { | ||
| return; | ||
| } | ||
| if rows.is_empty() { |
There was a problem hiding this comment.
Non-blocking (UX): "instance has been idle" is chosen when any finite sample exists in the last ~240 ticks, not from the current state. If the observation has since expired, the scrape is failing, or the instance restarted under the same id, the summary shows gen: - / "unknown" while this says it's idle. Basing it on the latest tick's sample, or using neutral wording ("no non-zero throughput in recent history"), would avoid over-claiming.
Summary
draw_detailincrates/rocm-dash-tui/src/ui/instance_detail.rs: scans the in-memory metrics history for the last few snapshot ticks where the instance was actually generating tokens (gen_tps > 0, finite), and renders them newest-first as a small table (time / tok/s / TTFT / TPOT). Shows a muted placeholder distinguishing "no throughput samples" from "no recent throughput — instance has been idle" depending on whether a real zero-throughput sample was actually observed. Repeated Held ticks of one scrape collapse to one row, aged fromobserved_at; when a shown row carries the held*, the panel title shows the sharedformat::HELD_LEGEND.activity_layout) and the launch args / env body keeps at leastmax(6, half the modal)rows (body_floor), so the panel shrinks (title saysnewest K) or collapses on short terminals instead of starving the body. At 80x24 the panel shows one row and the body keeps several launch-arg lines.Test plan
dash-42(@id:dash-instance-detail-recent-throughput,@requires-os:linuxonly, no GPU tag;dash.feature, step indash_steps.rs): opens instance detail for a scripted-metrics managed model and requires an "Ns ago <tok/s>" row. Runs at the harness's 80x24 terminal. The placeholders and short-terminal collapse are covered by render tests, not a scenario.cargo test -p rocm-dash-tui— all pass, includingui::instance_detailtests for the 80x24 body floor, held legend, age clamp, layout sweep and sibling-instance filtering. Breakingbody_floorback to 3 turns four of them red.cargo clippy -p rocm-dash-tui --all-targets -- -D warnings,cargo fmt --all -- --check— clean.comfyuitests under parallel load; they pass in isolation (85/85).