Skip to content

e2e: PTY-observed download/extraction spinner coverage, fix truncated progress display - #421

Merged
jussielo-amd merged 25 commits into
mainfrom
worktree-e2e-download-spinner-pty
Sep 30, 2026
Merged

jussielo-amd merged 25 commits into
mainfrom
worktree-e2e-download-spinner-pty

Conversation

@jussielo-amd

@jussielo-amd jussielo-amd commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Adds two E2E scenarios that drive a real rocm binary under a real PTY to prove cli_progress::AnimatedSpinner actually renders and clears correctly outside of its existing in-process unit tests:
    • download-progress-01: TheRock tarball SDK install — download progress, extraction ("Extracting …") frame, and both spinner lines cleared on exit.
    • comfyui-04: ComfyUI source-archive install — download progress only (this path has no separate extraction phase) and the spinner line cleared on exit.
  • Adds a shared PacedDownloadServer fixture (tests/e2e-cucumber/src/paced_download.rs): a loopback HTTP server that streams one named file in fixed-size, delayed chunks (falling back to normal ServeDir for everything else), so the PTY has a real chance to observe an intermediate, sub-100% frame instead of the whole transfer completing within one screen poll.
  • Adds two e2e-test-hooks-gated hooks needed to make the ComfyUI scenario hermetic and fast: ROCM_CLI_COMFYUI_SOURCE_ARCHIVE_URL_OVERRIDE (redirect the source-archive download to the fixture server) and ROCM_CLI_DISABLE_TORCH_RUNTIME_DEP_CHECKS (skip the real system package-manager dependency check that otherwise runs both after a successful SDK install and during rocm engines install vllm). Both compile out entirely in non-e2e-test-hooks builds, so production behavior is unaffected.
  • Documents the new override in docs/release-trust.md, alongside the existing TheRock base-override documentation.
  • Fixes a real production bug in cli_progress::Spinner: tightening the new e2e assertion to require an actual intermediate (non-0%, non-100%) progress frame reproducibly failed, which traced back to render_current truncating the whole assembled status line from the tail. On an ordinary 80-column terminal, once the byte count grew past a couple of characters (e.g. "0 B" → "1.5 MiB"), the trailing "(NN%)" got silently ellipsized away — hiding the download percentage for the entire rest of the transfer, with only the very first (0%) frame ever visible. Fixed by splitting the label (the file name, already shown elsewhere) from the byte-count/percentage suffix, and truncating only the label when the line overflows, so the percentage always stays visible. Also hardens the new assemble_status_line helper against an even narrower-terminal edge case where the suffix itself doesn't fit.
  • Two rounds of PTY-harness hardening in response to review feedback: made the @serial-tagged scenarios' negative "spinner line cleared" assertions robust against both truncation (matching a prefix instead of the full, truncated label) and the emulated terminal's zero-scrollback screen (growing PTY rows so a later summary can't scroll a spinner row out of view before the check reads it).

Fixes #368

Why: AnimatedSpinner only had in-process unit coverage before this — never run under a spawned subprocess — so a regression in its TTY detection, throttling, or line-clearing on Drop could ship unnoticed. This exercises both the tarball and ComfyUI download paths end to end against real fixture servers, and in doing so caught a genuine, independently-reproducible bug in the spinner's own rendering logic (described above).

Risk: low-to-medium. Most of the diff is test-only, but apps/rocm/src/cli_progress.rs's render_current/assemble_status_line change is a real behavioral fix shipped to production — it changes what gets truncated on narrow terminals during any download or long-running command that uses the spinner. Covered by unit tests (including two regression tests for both truncation edge cases) and the new e2e scenarios.

Known gaps

  • The dependency-install spinner added by the recent spinner work (apps/rocm/src/comfyui.rs's uv block) isn't exercised by this coverage — the ComfyUI fixture deliberately empties the dependency list so that code path never runs during these scenarios. Intentional scope limit, not an oversight; left for follow-up coverage.
  • download-progress-01's extraction-frame assertion is inherently timing-dependent: a 20ms poll against a real tar subprocess run. @serial (no concurrent CPU contention) and generous payload sizing mitigate this but don't eliminate it — it's the residual flake source to watch if this scenario ever becomes intermittent.

Test plan

  • cargo build --workspace --all-targets — clean
  • cargo clippy --workspace --all-targets -- -D warnings — clean
  • cargo test --bin rocm cli_progress — 15 passed
  • cargo test -p e2e-cucumber --lib — 122 passed
  • cargo xtask e2e -- -n "download-progress-01|comfyui-04" — both scenarios pass, run repeatedly, 11/11 steps each time
  • Full local suite (cargo test -p e2e-cucumber --test e2e, no filter — exercises cucumber's default 64-concurrent-scenario execution) run repeatedly with 0 unexpected failures, to confirm no contention-induced flake
  • prek run --all-files --no-group local-tools — clean

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Both progress assertions can pass on the initial 0% callback without observing transferred data.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 Medium severity

Open (2)
What changed in this PR

Adds PTY-based E2E coverage for download and extraction spinner behavior.

Changes:

  • Adds paced HTTP fixtures for observable progress.
  • Covers TheRock and ComfyUI spinner rendering and cleanup.
  • Adds feature-gated test overrides for hermetic execution.
File Description
apps/​rocm/​src/​comfyui.rs Adds the test-only source URL override.
apps/​rocm/​src/​main.rs Adds dependency-check bypass for E2E tests.
Cargo.lock Records the new dependency.
docs/​release-trust.md Documents the ComfyUI override.
tests/​e2e-cucumber/​Cargo.toml Adds futures.
tests/​e2e-cucumber/​features/​comfyui.feature Adds ComfyUI spinner scenario.
tests/​e2e-cucumber/​features/​download_progress_pty.feature Adds TheRock spinner scenario.
tests/​e2e-cucumber/​src/​lib.rs Exports the paced-download fixture.
tests/​e2e-cucumber/​src/​paced_download.rs Implements and tests paced HTTP downloads.
tests/​e2e-cucumber/​tests/​e2e.rs Manages paced-server lifecycle.
tests/​e2e-cucumber/​tests/​e2e/​comfyui_steps.rs Implements ComfyUI PTY steps.
tests/​e2e-cucumber/​tests/​e2e/​therock_steps.rs Implements TheRock PTY steps.
tests/​e2e-cucumber/​tests/​feature_naming.rs Registers the new feature prefix.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

jussielo-amd added a commit that referenced this pull request Sep 21, 2026
Copilot review comment on #421 pointed out that both PTY scenarios'
'intermediate download progress frame' assertions accepted the
unthrottled (0%) frame emitted before any bytes are read, so they
passed even if no real in-transfer frame ever rendered. Tightening
both to require a percentage strictly between 0% and 100%
(wait_for_screen_where over a percent that is neither (0%) nor
(100%)) turned up a real product bug: on an ordinary 80-column
terminal, Spinner::render_current truncated the whole assembled
'{frame} {label} {bytes}/{total} (pct%)' line from the tail once the
byte count grew past a couple of characters (e.g. '0 B' -> '1.5
MiB'), silently dropping the '(pct%)' suffix for the rest of the
transfer.

Fix: keep the byte-count/percentage suffix in its own field
(progress_suffix) and truncate the file name label instead when the
line doesn't fit -- the label was already printed in full earlier in
the command's output, so it's the safer thing to sacrifice, and the
percentage is the part a user actually watches move.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@jussielo-amd
jussielo-amd requested a balanced review from Copilot September 21, 2026 10:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The TheRock scenario’s clearing assertion cannot detect the truncated download spinner it is intended to verify.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
Resolved since last review (2)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The tarball final-screen assertion can pass after later output scrolls an uncleared spinner off-screen.

Review effort: Balanced
Findings: None

Resolved since last review (1)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Spinner-clearing checks run after the row has scrolled off-screen

tests/​e2e-cucumber/​tests/​e2e/​therock_steps.rs:479

screen_text() contains only the current 24-row VT screen. After extraction, this command prints roughly 21 summary lines, and its long URL/path lines wrap at the fixed 80-column width, so the spinner row is guaranteed to scroll out even if Spinner::clear is broken. These negative checks therefore cannot verify clearing. Capture the PTY transcript/erase sequence, or synchronize an assertion before the subsequent summary can scroll the row away.

@jussielo-amd

jussielo-amd commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Addressing Copilot's "Needs a closer look" finding from this review ("Spinner-clearing checks run after the row has scrolled off-screen"): confirmed real. The PTY's vt100::Parser has 0 lines of scrollback, and the tarball install's ~20-line summary output was enough to scroll the download/extraction spinner's row off the top of the default 24-row screen before assert_spinner_lines_cleared ever read it — a scrolled-off row and a genuinely cleared one are indistinguishable to screen_text(), so the check could pass even if Spinner::clear were broken.

Fixed in 8783124:

  • Added TuiSession::grow_rows, which grows only the row count (not columns, unlike the existing use_detail_size) — widening columns too would stop the label from truncating, undermining the 80-column truncation behavior this scenario exists to exercise.
  • Called it right after spawning the tarball install session.

Verified two ways: dumped the screen at the assertion point and confirmed the whole session's output (from its first line onward) now fits with rows to spare; and temporarily neutered Spinner::clear() to confirm the assertion now genuinely fails when clearing is broken (it didn't catch this before the fix).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The production fix and both observable download paths have focused regression and end-to-end coverage.

Review effort: Balanced
Findings: None

@jussielo-amd
jussielo-amd marked this pull request as ready for review September 21, 2026 12:56
@jussielo-amd
jussielo-amd requested a review from a team as a code owner September 21, 2026 12:56

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 8783124

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Adds two PTY-driven Gherkin scenarios that observe the download/extraction progress spinner through a real pseudo-terminal, a paced chunked-HTTP fixture server to make intermediate frames observable, and — not purely test work — two production fixes to apps/rocm/src/cli_progress.rs (keep the byte/percentage suffix intact when the label overflows; never exceed max_width when the suffix alone overflows) plus two e2e-test-hooks-gated override accessors. No blocking findings. Verified: mutated each of the two new production branches separately in a scratch copy and confirmed each new unit test reddens on its own assertion (whole-line-truncation revert → the suffix-intact test fails; removing only the narrow-terminal guard → the max-width test fails, while the other still passes); the PTY scenarios themselves were not executed here and the full suite was not run. Check state at review: 19 success, 2 skipped, 0 failure. Blocking: 0 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

None.

Notes on the specific risks checked, since this PR's value rests entirely on whether the new assertions discriminate:

  • Vacuous passes. Every positive PTY assertion is a contains on rendered text, so an empty screen fails it. wait_for_screen_where/wait_for_screen return Err on timeout, on reader-thread panic, and on the child exiting early, so a missing binary or a process that dies does not pass. wait_for_exit asserts exit code 0, not merely that the process exited. TuiSession::spawn surfaces an OS spawn failure as an error.
  • Negative assertions vs. scrolling. The vt100 parser is built with zero scrollback, so a scrolled-off row is indistinguishable from a cleared one. therock_steps.rs handles this explicitly by growing the PTY to 60 rows before the summary prints. For the ComfyUI scenario the arithmetic holds independently: only lines printed after the spinner row can push it off a 24-row screen, and that path prints ~10, so the row cannot scroll away — see non-blocking #1 for the fragility this leaves behind.
  • Negative assertion vs. truncation. assert_spinner_lines_cleared correctly checks a 30-character prefix of the download label rather than the full string, because the live line is truncated on an 80-column terminal; the full string would have matched nothing either way. The extraction label carries no suffix, is 59 columns wide, and so is checked in full — correct. The permanent tarball: <file> line printed at apps/rocm/src/therock.rs:2611 does not contain the Downloading … prefix, so it cannot defeat the check.
  • Readiness waits. No wait in this diff is satisfied before the action it gates: each is a positive wait for output that only the action produces, and each fails loudly if it never appears.
  • @serial is real, not decorative. I initially concluded the tag was inert because the repo's own tag parser (tests/e2e-cucumber/src/expectation.rs) does not recognise it. That was reviewer error: @serial is consumed by cucumber-rs's default which_scenario, which maps it to ScenarioType::Serial; the repo sets only max_concurrent_scenarios and never overrides which_scenario, and the repo parser silently ignores unknown tags. The tag works as the feature-file comments claim. See non-blocking #4 for the one-line fix that stops this recurring.
  • Coverage vs. the previous version (focus g). The rename of format_download_progress to the private format_progress_suffix carries every prior assertion forward. One old assertion — that the rendered string starts with the caller's prefix — is dropped, and that vacation is correct, not a regression: the function no longer takes a prefix, so the assertion no longer has a subject; the prefix/suffix composition is now pinned structurally by assemble_status_line and by two new tests. The function had no other callers at the base, so nothing lost coverage elsewhere.
  • Sign-off and content. All 9 commits in the range carry exactly one Signed-off-by: trailer each (checked line-by-line from the raw commit objects). No internal names, hostnames, gateways, cluster names, registry paths or internal document links in the diff. No prompt-injection content found anywhere in the checkout.

Non-blocking

  1. tests/e2e-cucumber/tests/e2e/comfyui_steps.rs:540 — the "no ComfyUI download spinner line" check is safe today only because ~10 lines print after the spinner; the sibling tarball scenario protects the identical assertion explicitly with grow_rows(60). Any future growth of the ComfyUI completion report would silently turn this into a tautology with no test failure to warn you. Call grow_rows here too, or state the invariant in a comment.
  2. apps/rocm/src/cli_progress.rs:65 — set_label now resets progress_suffix to None, and nothing tests it; a regression there would leave a stale byte count glued to an unrelated message. One unit test (set_progress, then set_label, then assert the suffix is gone) closes it.
  3. tests/e2e-cucumber/tests/e2e/therock_steps.rs — &format!("Downloading {CURRENT_TARBALL}")[..30] is a raw byte slice with a magic length; it panics if the fixture file name is ever shortened or gains a non-ASCII character. A named constant with a one-line note on why 30, or a char-boundary-safe slice, removes the trap.
  4. tests/e2e-cucumber/src/expectation.rs — every scenario tag the repo owns is documented on ScenarioDecl, and @serial is the only tag in the feature files that is not, because cucumber-rs owns it. One line there saying so would have prevented the wrong conclusion I reached above, and will otherwise invite the same conclusion from the next reader.
  5. The dependency-install spinner added by the recent spinner work (apps/rocm/src/comfyui.rs:1712) is the one spinner this coverage does not reach — the ComfyUI fixture deliberately empties the dependency list so the uv block never runs. Worth naming as a known gap in the PR text rather than leaving it implicit. Separately, the PR title reads as test-only, but the diff changes production rendering in cli_progress.rs and adds a gated early-return in main.rs; the description should say so, since a maintainer skimming "e2e: … coverage" would not expect behaviour changes. The extraction-frame assertion also remains inherently timing-dependent (a 20 ms poll against a real tar run) — @serial plus the payload sizing mitigate it, but it is the residual flake source to watch.

@jussielo-amd jussielo-amd changed the title e2e: PTY-observed download/extraction progress spinner coverage e2e: PTY-observed download/extraction spinner coverage, fix truncated progress display Sep 22, 2026
@jussielo-amd

Copy link
Copy Markdown
Collaborator Author

Addressed all 5 non-blocking items from review #5270510051 in 9be2f78 + PR title/description:

  1. comfyui_steps.rs — grew the ComfyUI PTY to 60 rows before install, mirroring the tarball scenario's fix, so a future-longer completion report can't turn the "cleared" check into a tautology the way it could have before.
  2. cli_progress.rs — added a unit test proving set_label clears a stale progress_suffix.
  3. therock_steps.rs — replaced the magic [..30] byte slice with a named, documented DOWNLOADING_PREFIX_LEN constant and a panic-safe .get(..).unwrap_or(...) lookup.
  4. expectation.rs — documented that @serial is intentionally outside ScenarioDecl/from_tags (it's a cucumber-rs runner concern, not an expectation-resolution one).
  5. PR title now names the production fix rather than reading as test-only; description gained a "Known gaps" section calling out the untested comfyui.rs dependency-install spinner path and the residual timing-dependence of the extraction-frame assertion.

All verified: full workspace build/clippy/test, targeted + full-suite (repeated, contention-inclusive) e2e runs, and prek, all clean.

jussielo-amd added a commit that referenced this pull request Sep 23, 2026
Copilot review comment on #421 pointed out that both PTY scenarios'
'intermediate download progress frame' assertions accepted the
unthrottled (0%) frame emitted before any bytes are read, so they
passed even if no real in-transfer frame ever rendered. Tightening
both to require a percentage strictly between 0% and 100%
(wait_for_screen_where over a percent that is neither (0%) nor
(100%)) turned up a real product bug: on an ordinary 80-column
terminal, Spinner::render_current truncated the whole assembled
'{frame} {label} {bytes}/{total} (pct%)' line from the tail once the
byte count grew past a couple of characters (e.g. '0 B' -> '1.5
MiB'), silently dropping the '(pct%)' suffix for the rest of the
transfer.

Fix: keep the byte-count/percentage suffix in its own field
(progress_suffix) and truncate the file name label instead when the
line doesn't fit -- the label was already printed in full earlier in
the command's output, so it's the safer thing to sacrifice, and the
percentage is the part a user actually watches move.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@jussielo-amd
jussielo-amd force-pushed the worktree-e2e-download-spinner-pty branch from 20dbbdb to af6bdee Compare September 23, 2026 08:09
@jussielo-amd
jussielo-amd requested a balanced review from Copilot September 23, 2026 08:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The tarball spinner test remains non-hermetic because its OpenMPI preflight may invoke apt/sudo.

Review effort: Balanced
Findings: None

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 20dbbdb

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Adds two PTY-driven Gherkin scenarios that observe the download/extraction progress spinner through a real pseudo-terminal, a paced chunked-HTTP fixture server that makes intermediate frames observable, and production changes to apps/rocm/src/cli_progress.rs (split the byte/percentage suffix out of the label so truncation can never eat it, plus a narrow-terminal clamp) alongside two e2e-test-hooks-gated override seams. Since the last round it also lands the five remediation items and one further production fix (a doubled space in the narrow-terminal fallback); no blocking findings. Verified: ran the cli_progress unit module in a scratch copy (16 passed) and then mutated each production branch separately there — reverting set_label's suffix reset reddens only set_label_clears_a_stale_progress_suffix; reintroducing the doubled space reddens only assemble_status_line_fallback_does_not_double_the_space_before_suffix; disabling the narrow-terminal guard reddens the two width tests; reverting to whole-line truncation reddens only keeps_the_progress_suffix_intact… — so each branch is pinned by its own assertion, not just by a wholesale revert. I also confirmed from source that "Downloading therock-dist-linux" (the exact 30-byte prefix) and "Extracting {tarball}" and "Fetching ComfyUI source archive" each occur exactly once in the product, as spinner labels only, so none of the negative checks is a tautology; that wait_for_exit asserts exit code 0 and wait_for_screen_where returns Err on timeout, child exit and reader panic, so nothing passes on an empty screen; that the only runtime source of "%)" reachable from these two commands is the spinner suffix itself; that the vt100 parser has zero scrollback while grow_rows(60) comfortably covers the ~20 lines printed after the spinner; that @serial is honoured by cucumber-rs's default which_scenario and @requires-os:linux resolves to Skip; that e2e-test-hooks is off by default with no --all-features anywhere in CI, so both new override seams are compiled out of release builds; and, measuring it rather than trusting the comment, that high-entropy filler really is gzip-incompressible (20 MB, ratio 1.0003), which is what keeps the pacing real. All 16 commits carry exactly one sign-off; leak and injection scans over the diff are clean. The merge base (93677c8) is behind the prw-base tip (3ecf484) by four commits, but the resulting diffs are byte-identical, so scope is unaffected. Not run here: the e2e/PTY scenarios themselves and the full workspace suite. I could not check the PR title and description text, since this review never contacts GitHub — the author's claim that item 5 was addressed there is unverified by me. Checks at this head: 18 success, 2 skipped. Blocking: 0 · Non-blocking: 5. One thing to flag about timing: this review was carried out against the commit named in the heading above, and the branch was force-updated onto a newer base while it was running. The branch's own commits look unchanged by that update, but everything above was assessed before it, so it speaks for the commit it names rather than for whatever is at the tip now. Checks at the new tip stood at 18 success, 2 skipped and 1 still running when this was posted.

🚫 Blocking (must fix before merge)

None.

Two things are worth stating plainly, because this PR's whole value rests on whether the new assertions discriminate.

  • The remediation round was weighed as original code, not as review answers. Item 2 (set_label clearing a stale suffix) and the unprompted item in 50fa83d (dropping the doubled space) are both real production edits, and both were mutation-tested above: each fails exactly one test and leaves the others green. Item 3 replaced the raw byte slice [..30] with DOWNLOADING_PREFIX_LEN plus .get(..).unwrap_or(&downloading), which is genuinely panic-safe — a shortened or non-ASCII fixture name now degrades to checking the full string rather than aborting. Item 7's shim dedup (write_gpu_probe_shim) is byte-identical to both previous inline copies. Item 6's rand swap replaces a hand-rolled PRNG with a seeded StdRng and honestly documents in the same comment that rand does not promise algorithm stability across versions, so a future bump could change the fixture bytes — that caveat is accurate, not hedging.
  • Where the Drop-time clear is actually proven. Spinner::clear on Drop is covered, but not by the half of the assertion a reader would assume — see non-blocking #1. The ComfyUI scenario is what carries it for a download spinner, and the extraction half of the tarball assertion carries it there; both were traced through the real output sequence rather than assumed.

Non-blocking

  1. tests/e2e-cucumber/tests/e2e/therock_steps.rs:1300 — the "Downloading therock-dist-linux" half of assert_spinner_lines_cleared cannot fail for a broken Drop-time clear: the extraction spinner starts on the same row and its first repaint issues Clear(CurrentLine), erasing the download line whether or not Spinner::clear ran. The extraction half and the ComfyUI scenario do cover that defect, so nothing is untested — but the step's own comment reads as if both halves prove it. One sentence saying the download half guards in-place repaint (not Drop-clear) would stop the next reader drawing the same wrong conclusion I nearly did.
  2. apps/rocm/src/cli_progress.rs:208 — assemble_status_line takes suffix: Option<&str> but its narrow-terminal branch silently depends on the caller's suffix carrying its own leading space; a future second caller passing a space-less suffix would glue frame and suffix together with no test to catch it. The invariant is pinned today only because format_progress_suffix is the sole producer.
  3. apps/rocm/src/cli_progress.rs:123 — the Err(_) arm of render_current still emits an unbounded, untruncated line when terminal::size() fails. This is preserved base behaviour, not a regression, and it is the one rendering path with no coverage at any level.
  4. tests/e2e-cucumber/tests/e2e/therock_steps.rs:1229 and comfyui_steps.rs:1042 — grow_rows(60) is issued immediately after TuiSession::spawn, with no synchronisation point, so there is a theoretical window where the child could emit more than 24 lines before the resize lands. Not reachable at real process-startup timings, but the assumption is unstated.
  5. The extraction-frame assertion remains inherently timing-dependent (a 20 ms poll against a real tar over a ~20 MB archive); @serial plus the payload sizing give it roughly a 90-poll margin on the download side, but it stays the residual flake source to watch. Separately, apps/rocm/src/comfyui.rs:1712's dependency-install spinner is still the one spinner this coverage deliberately does not reach.

@jussielo-amd

Copy link
Copy Markdown
Collaborator Author

Addressed the non-blocking items from the automated review at commit 20dbbdbd:

  • Let Lemonade auto-select its llama.cpp backend #1 (download-half of assert_spinner_lines_cleared doesn't itself prove Drop-time clear) — fixed: reworded the comment to say explicitly that the extraction assertion (and the ComfyUI scenario) are what actually cover that, not this check.
  • Enable native-certs for ureq across all crates #2 (assemble_status_line's suffix-leading-space invariant unenforced) — fixed: added a debug_assert! so a future space-less caller fails loudly in tests instead of silently gluing frame/suffix together.
  • Migrate Python environment management from pip to uv #3 (render_current's Err(_) fallback emits an unbounded, untruncated line) — fixed: it now shares assemble_status_line with an 80-column fallback width, same as every other path.
  • Fix CI: prek hooks, headless/flaky test fixes, and drop vendored Codex #4 (grow_rows(60) issued right after spawn with no sync point) — documented rather than eliminated: added a comment stating the timing assumption explicitly (fork/exec + first repaint take longer than the resize call), matching your own "not reachable at real timings" assessment.
  • Speed up CI #5 (extraction-frame timing dependence, ComfyUI dependency-install spinner not covered) — left as-is; both are already called out in the PR description's "Known gaps" as accepted scope limits, not regressions.

Also caught and fixed an off-by-one in assemble_status_line itself (exact-fit boundary produced a doubled space) via a separate review pass, plus a couple of minor cleanups (deduped two near-identical PTY step assertions onto TuiSession, resolved the ComfyUI source URL once instead of twice). All commits since your review: 3ef8bb8d, d103d4dc, 3b03f6d9, 6474f5a0. Full workspace build/clippy/fmt clean, and both download-progress-01/comfyui-04 PTY scenarios re-verified passing on the latest commit.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice PR overall. The core fix in cli_progress.rs (assemble_status_line truncating the label instead of the whole line, so the download percentage suffix always survives) is exactly right, has a clear regression test proving the original bug, and the two new e2e-test-hooks production hooks compile out to hardcoded safe defaults outside test builds. CI is green including the new E2E scenarios.

I found a few issues in the new test infrastructure worth fixing before merge, left as inline comments. None of them touch the production fix itself.

  • build_gzip_tarball shells out to tar synchronously from inside async fn given-steps. That blocks a tokio worker thread for the duration of a ~20MB tar invocation, and with the mock lane's default concurrency that can starve other in-flight scenarios' async work, similar in spirit to the CPU contention the "Known gaps" section already calls out for a different reason.
  • assert_spinner_lines_cleared's prefix truncation falls back to the full untruncated string on a non-char-boundary split instead of failing loudly, which would silently defeat the exact regression the assertion exists to catch.
  • The "reuse existing source folder" path in comfyui.rs::install() records a fresh call to comfyui_source_archive_url() as the manifest's source_url, not the URL that actually produced the on-disk folder. Invisible in production today (hardcoded constant), but a latent provenance bug under the test override.
  • paced_download.rs's own unit tests only exercise single-segment routes, never the multi-segment routes both real E2E scenarios actually register.

Comment thread apps/rocm/src/comfyui.rs Resolved
jussielo-amd added a commit that referenced this pull request Sep 24, 2026
Copilot review comment on #421 pointed out that both PTY scenarios'
'intermediate download progress frame' assertions accepted the
unthrottled (0%) frame emitted before any bytes are read, so they
passed even if no real in-transfer frame ever rendered. Tightening
both to require a percentage strictly between 0% and 100%
(wait_for_screen_where over a percent that is neither (0%) nor
(100%)) turned up a real product bug: on an ordinary 80-column
terminal, Spinner::render_current truncated the whole assembled
'{frame} {label} {bytes}/{total} (pct%)' line from the tail once the
byte count grew past a couple of characters (e.g. '0 B' -> '1.5
MiB'), silently dropping the '(pct%)' suffix for the rest of the
transfer.

Fix: keep the byte-count/percentage suffix in its own field
(progress_suffix) and truncate the file name label instead when the
line doesn't fit -- the label was already printed in full earlier in
the command's output, so it's the safer thing to sacrifice, and the
percentage is the part a user actually watches move.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The non-e2e-test-hooks accessor isn't absent, it's a cfg(not(...))
variant that unconditionally returns the hardcoded URL. Reword so the
safety property is stated accurately.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
xorshift_payload duplicated PRNG logic the workspace already depends
on for the same "deterministic filler bytes" purpose, and a separate
hand-rolled xorshift already exists in rocm-dash-daemon's demo
generator. Renamed to deterministic_payload and rewritten on top of
rand::rngs::StdRng, seeded the same way, so the paced-download
fixtures stay deterministic without a third bespoke implementation.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The 8-line shell script answering probe_comfyui's post-install GPU
check was duplicated verbatim across two scenario fixtures. Extracted
into write_gpu_probe_shim so a future probe-schema change only needs
updating in one place.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
assemble_status_line took the label-truncation branch even when the
reserved frame+suffix width exactly matched max_width, truncating the
label to nothing while still emitting both its own literal space and
the suffix's leading space -- one column narrower and the fallback
branch produced a single space instead. Route the exact-fit boundary
through that same fallback.

render_current's terminal::size() Err arm built its status line by
hand with no truncation at all, the one rendering path with no width
bound or test coverage; it now shares assemble_status_line with an
80-column fallback width like every other path.

Also folds Spinner's separately-mutated label/progress_suffix fields
into a single SpinnerText enum, so "a plain message never carries a
stale byte-count suffix" is structural rather than a convention every
setter has to remember, and asserts the suffix-has-a-leading-space
invariant assemble_status_line's narrow fallback depends on.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
assert_spinner_lines_cleared's download-line check reads as if it
proves Spinner::clear runs on Drop, but the extraction spinner's first
repaint overwrites the same row with Clear(CurrentLine) regardless --
the extraction assertion and the ComfyUI scenario's equivalent check
are what actually cover that. Note it explicitly so the next reader
doesn't draw the same wrong conclusion.

Also documents the unstated timing assumption behind issuing
grow_rows(60) right after TuiSession::spawn with no synchronization
point: it relies on fork/exec and the first repaint taking longer than
the resize call.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
therock_steps.rs and comfyui_steps.rs each hand-rolled the identical
"wait for an intermediate download frame" and "wait for clean exit"
step bodies, differing only in a panic-message context string. Move
both onto TuiSession as assert_intermediate_download_progress_frame
and assert_exits_cleanly, parameterized by that context, so the two
step files call a shared helper instead of drifting independently.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
install() called comfyui_source_archive_url() a second time when
recording the ComfyUI manifest, after download_and_extract_source had
already resolved it once for the actual download. Have that function
return the URL it used instead, so the manifest always records exactly
what was (or would have been) fetched, not a second, independent
resolution of the same accessor.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
- build_gzip_tarball now runs tar via spawn_blocking instead of
  blocking a tokio worker thread synchronously from an async
  given-step.
- The tarball spinner-cleared assertion now fails loudly instead of
  silently degrading to an untruncated-label check if
  DOWNLOADING_PREFIX_LEN ever lands off a UTF-8 char boundary.
- ComfyUI's "reuse existing source folder" path now records the
  source_url from the prior manifest on disk instead of a fresh call
  to comfyui_source_archive_url(), so the recorded provenance matches
  what actually produced the folder.
- Added a paced_download unit test covering a multi-segment route,
  matching the real callers' route shapes.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Reuse-source-folder path now falls back to the current source URL
whenever the on-disk manifest can't be read or parsed, instead of
propagating the error and aborting install. Previously install was
the one command that recovered from a broken manifest; the prior
cleanup regressed that by making a failed load_manifest fatal.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Extracts the reuse-existing-source-folder URL lookup into
reused_source_url() and adds unit tests for the no-manifest,
valid-manifest, and unparseable-manifest cases, closing the gap
where the unreadable-manifest branch had no regression coverage.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Non-blocking items from the automated review at 36a59e9: add an
exact-output test for the ordinary label-fits path (the existing
tests only checked contains/width, so a dropped separator space or
an off-by-one label_budget slipped through undetected), drop the
unreachable empty-suffix guard in assemble_status_line, tighten two
doc comments, and note that comfyui-04's short label never exercises
the truncation fix.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@jussielo-amd
jussielo-amd force-pushed the worktree-e2e-download-spinner-pty branch from dfc5b56 to 683c237 Compare September 29, 2026 05:27
@siloteemu

Copy link
Copy Markdown
Contributor

Round 4 posted — the review above has been updated in place for 683c2374.

@jussielo-amd
jussielo-amd added this pull request to the merge queue Sep 29, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Sep 29, 2026
@jussielo-amd
jussielo-amd added this pull request to the merge queue Sep 30, 2026
Merged via the queue into main with commit b0d598d Sep 30, 2026
30 of 31 checks passed
@jussielo-amd
jussielo-amd deleted the worktree-e2e-download-spinner-pty branch September 30, 2026 06:09
r0x0r added a commit that referenced this pull request Oct 13, 2026
…child-pipes

Two conflicts, one of them a whole-file collision.

crates/rocm-core/src/process.rs: #599 extracted a module of the same
name out of lib.rs, so main and this branch each created a different
file at that path. Resolved as a union -- the two have zero name
overlap, main's being spawn/terminate primitives and `FileLock`, this
branch's being `run_with_timeout` and its `Capture` machinery, which is
exactly the module main just created for this kind of thing. Imports
merged (main's narrower `use std::process::Command` dropped for this
branch's wider one; `anyhow::bail` widened to unconditional and the
windows-only duplicate removed), and both test modules joined -- 17
`process::` tests now, main's three `FileLock` cases included.

docs/architecture.md: main added net.rs/paths.rs/process.rs bullets.
Kept them, folding `run_with_timeout` into the process.rs entry. The
previous merge had reinstated the claim that this is "the one bounded
child-process execution helper the apps share", which 6533327 removed on
purpose; rominf caught that, and it is not reinstated here. The bullet
now says plainly that other capture helpers still join their readers
unbounded.

Also fixes jussielo-amd's blocking finding: the wait loop's two `?`
early returns -- `try_wait` and the post-kill `wait` -- returned before
`take()` was reached, so `abandoned` was never set and each reader
thread kept appending to a buffer nobody would ever read, contradicting
`Capture`'s own documented bound. Both paths now abandon first.

Verified on Linux: clippy --workspace --all-targets -D warnings exit 0;
cargo test --workspace --all-targets 32 groups, the only failure being
the known `animated_spinner_keeps_ticking` load flake from #421, which
passes 3/3 in isolation.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
r0x0r added a commit that referenced this pull request Oct 13, 2026
…child-pipes

Two conflicts, one a whole-file collision.

crates/rocm-core/src/process.rs: #599 extracted a module of the same
name out of lib.rs, so main and this branch each created a different
file at that path. Resolved as a union -- zero name overlap between
them, main's being spawn/terminate primitives and `FileLock`, this
branch's being `run_with_timeout` and its `Capture` machinery, which is
exactly the module main just created for this kind of thing. Imports
merged (main's narrower `use std::process::Command` dropped for the
wider one; `anyhow::bail` widened to unconditional and the windows-only
duplicate removed), both test modules joined -- 17 `process::` tests,
main's three `FileLock` cases included.

docs/architecture.md: kept main's net.rs/paths.rs/process.rs bullets,
folding `run_with_timeout` into the process.rs entry. The previous merge
had reinstated the claim that this is "the one bounded child-process
execution helper the apps share", which 6533327 removed on purpose;
rominf caught that, and it is not reinstated. The bullet now says
plainly that other capture helpers still join their readers unbounded.

Also fixes jussielo-amd's blocking finding: the wait loop's two `?`
early returns -- `try_wait` and the post-kill `wait` -- returned before
`take()` was reached, so `abandoned` was never set and each reader
thread kept appending to a buffer nobody would ever read, contradicting
`Capture`'s own documented bound. Both paths now abandon first.

Verified on Linux: clippy --workspace --all-targets -D warnings exit 0;
cargo test --workspace --all-targets 32 groups, the only failure the
known `animated_spinner_keeps_ticking` load flake from #421, which
passes 3/3 in isolation.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
jussielo-amd pushed a commit to jussielo-amd/rocm-cli that referenced this pull request Oct 16, 2026
…g timeouts (ROCm#526)

* fix(rocmd): drain child pipes concurrently so output size stops faking timeouts

Captured output was collected by waiting for the child to exit and only
then calling `wait_with_output`, the one call that drains the pipes. The
pipes are bounded by the OS (~64KiB on Linux, less on Windows), so a
child writing more than that blocks in `write` and never exits, and the
drain is never reached. The only way out was the timeout, which meant it
measured how much the child printed rather than how long it took, and
killed successful commands mid-flight.

Reachable two ways: the `amd-smi static -a -g all --json` probe runs
under a 2s budget and clears a buffer easily on a multi-GPU host, so GPU
telemetry reported `amd_smi_available: false` with a spurious timeout
note into `bridge-snapshot`, the `examine_snapshot`/`bridge_snapshot`
tools and the automation watcher; and an MCP-driven `install sdk` prints
far more than a buffer, so it was killed partway, leaving a partial SDK.

Three byte-identical copies of the loop existed -- `rocmd`'s
`run_command_with_timeout` and `run_process_with_timeout`, and a third in
`rocm`'s `main.rs` -- so they are replaced by one
`rocm_core::process::run_with_timeout` following the reader-thread
pattern already in `examine.rs`. Both apps already depended on
`rocm-core`, so no new crate edge is introduced.

Also makes the timeout bound the call. Killing the child does not close
the pipe when it forked a grandchild that inherited the write end, so
collection blocked until that grandchild exited -- a 1s timeout took 30s
against `sh -c "sleep 30"`, because dash forks rather than execs. The
readers now publish into a shared buffer and are abandoned after a short
grace instead of joined.

Covered at both levels: unit tests for a child flooding both pipes and
for a timeout that must return on schedule, and scenario examine-20,
which drives `rocmd bridge-snapshot` against a stub `amd-smi` emitting
~150KiB. The scenario needs no GPU -- the stub is reached through the
`PATH` fallback -- so it runs on the mock lane rather than being stranded
on a multi-GPU host.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(e2e): isolate HOME so examine-20 actually reaches its stub amd-smi

examine-20 asserted it was hermetic, and it was not. `resolve_amd_smi_binary`
tries `default_data_dir()/runtimes/registry` and then the home fallbacks
before `PATH`, and both resolve through `runtime_home_dir()`, which reads the
real `$HOME` via `BaseDirs::new()` rather than `ROCM_CLI_DATA_DIR`.
`isolate_env` sets the three `ROCM_CLI_*` dirs plus the shared caches but
never `HOME`, so on a Linux host carrying a managed SDK under `~/.rocm` the
real `amd-smi` ran and the stub never did -- and the scenario then failed on
the size assertion, blaming the pipe-drain defect for an environment problem.

Point `HOME` at the isolated root for this one `rocmd bridge-snapshot`
invocation. Scoped to the step rather than added to `isolate_env`, because
other scenarios deliberately inherit the real HOME/XDG environment for their
PTY defaults and would change behaviour if it moved.

Both comments claimed the empty `<data>/runtimes` registry was sufficient on
its own, and the step's went further and said HOME was already isolated.
Neither held; both now describe the mechanism that does.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(process): pin the drain grace under any shell, and stop a read error passing for EOF

The timeout test only guarded anything where `/bin/sh` is dash. `sh -c "sleep
30"` leaves a grandchild under dash, which forks, but not under bash, which
execs the sleep into the shell itself -- so on a host where sh is bash, an
unbounded reader wait passed the test in 1.00s. Every grandchild case now
backgrounds explicitly, so a grandchild exists under either shell, and runs
under each shell present on the host.

Nothing covered the drain grace itself: taking the output the moment the child
exits passed all three existing tests, because a reader started at spawn has
already drained a normal child's output before the 50ms wait poll even notices
the exit. Output that reaches the pipe after the child is gone is the only way
to ask for that case on demand, and is now a test, alongside one for a child
that exits 0 while a background grandchild holds the pipe.

A read error was treated as EOF, so a failed pipe came back as a short `Ok`.
`wait_with_output`, which this replaced, retried `Interrupted` and returned
anything else; that is restored, and `run_with_timeout` reports it rather than
handing a caller truncated output to parse as JSON.

Also, each found by review of this branch:

- examine-20 asserted only the first amd-smi probe, despite being named for the
  telemetry surviving. A failed `monitor` probe still reports
  `amd_smi_available: true` with a full `static_snapshot`, so the scenario
  passed against half its own symptom; it now asserts both probes and a null
  note, and keeps rocmd's stderr for the failure message.
- The snapshot note rendered only `error.to_string()`, so a probe killed at its
  timeout was reported as `failed to launch amd-smi ...`. It now uses
  `{error:#}` and carries the cause, and the scenario comment says what the note
  actually said rather than calling it a timeout note.
- docs/architecture.md still placed command-timeout plumbing in rocmd's
  `common.rs`; it moved to `rocm_core::process` in this change.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(process): bound the child output a timed-out command quotes

`run_with_timeout`'s timeout message quoted all of the child's captured
stderr -- or all of stdout when stderr was empty -- with no limit, and
this branch's `{error:#}` note (added so the note names the timeout
rather than the launch) renders that whole chain into the bridge
snapshot. The drain fix makes it worse, because more output is now
captured than before. The note is not a terminal message: the GPU-metrics
watcher copies it into the automation-event and audit JSONL files every
60s and both append helpers rewrite the whole file, so an amd-smi that
prints a snapshot's worth of JSON and then stalls past its 2s budget grew
both logs by that much per minute, each write costing more than the last.
Reproduced at 151,369 characters.

Bounded inside `run_with_timeout` rather than at the note in `common.rs`.
The quote is assembled there, that is the only place that knows those
characters are raw child output rather than something we wrote, and all
four callers -- both `common.rs` captures, the daemon sandbox and
`apps/rocm` -- inherit the bound instead of each needing its own. The
truncation follows the two existing in-tree precedents,
`examine.rs::truncate_to_chars` and `host_gpu.rs::truncate_diagnostic_line`:
head, sliced on char boundaries, with a visible `...[truncated]` marker.

Deliberately not covered: `capture_amd_smi_json`'s own non-zero-exit
`bail!` quotes all of stderr too. That one predates this branch -- it is
its own outermost context, so `.to_string()` already carried it unbounded
-- and belongs with the follow-up for the remaining unbounded capture
helpers rather than here.

The rest of the review round:

- e2e `examine-21` pins what the changed note now claims, end to end: a
  stub amd-smi that prints ~150KiB and then stalls must leave
  `amd_smi_available: false` and a note that names the probe, names the
  timeout, says it was shortened, and stays under 4000 characters.
  Reverting `{error:#}` to `.to_string()` fails it on the timeout claim;
  removing the bound fails it on the marker.

- Windows now runs the hazard tests instead of only the trivial one. A
  batch-file fixture gives it a both-pipes flood (cmd has no `yes`, and a
  script file avoids cmd's quote-stripping rules becoming the thing under
  test), and a flood-then-`ping -n 30` case that covers the timeout, the
  bound, and the grandchild in one: killing `cmd` does not kill `ping`,
  which keeps the inherited write ends open. The success case now asserts
  stderr on Windows too -- `1>&2` is a cmd redirection like any other, and
  skipping it left the second pipe unchecked on that platform.

- Three comments corrected against the code: the snapshot feeds the
  `gpu_snapshot` MCP tool, not the `examine_snapshot` sandbox tool, which
  returns an `ExamineSummary` only; only the timeout case was misnamed by
  the outer context, since the parse and non-zero-exit failures are
  already their own outermost context; and the isolated HOME note lives on
  `pty_env`, not `isolate_env`.

- `docs/architecture.md` no longer calls this the only bounded helper the
  apps share. `therock.rs::wait_with_output_bounded`, `examine.rs::run_raw`
  and `host_gpu.rs`'s two capture helpers still join their readers with no
  time limit; the doc now names them and says converging them is follow-up
  work.

Refs: ROCMAI-448

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(core): bound the abandoned reader and test the result assembly

Addresses the review on ROCm#526.

`run_with_timeout` abandons a reader once `DRAIN_GRACE` expires, which is
what stops a grandchild holding the pipe from deciding how long a 2s probe
takes. But the abandoned thread kept appending to a buffer that `take()` had
already emptied and that nothing would read again, so in `rocmd` -- a daemon
that probes amd-smi every 60s -- each such call grew an allocation for as
long as the grandchild lived. `take()` now marks the capture abandoned under
the buffer's own lock, and the reader drains to EOF and discards from that
point, so the buffer provably cannot grow after `take()` returns.

Discarding rather than dropping the read end is deliberate: closing the pipe
would send the still-writing grandchild EPIPE/SIGPIPE, which for the
background helpers a sandboxed installer legitimately leaves behind trades a
leak we can bound for a behaviour change we cannot predict. The thread still
lives until the grandchild closes the pipe; that is now stated in `Capture`'s
docs rather than left to be discovered.

The post-wait result assembly moves into `assemble_output`, which makes its
branches reachable from a unit test for the first time: a read failure on
either stream is reported rather than returned as truncated output, a timeout
outranks a read failure caused by the kill, and the timeout quote prefers
stderr, falls back to stdout, and says "no output" when there was none. Every
previous timeout fixture wrote to stderr only, so the fallback chain could
have been reordered unnoticed.

`truncate_to_chars` was a byte-identical copy of `examine.rs`'s; that one is
now `pub(crate)` and shared. Its doc claimed `host_gpu.rs`'s
`truncate_diagnostic_line` used the same marker -- it appends `...`, not
`...[truncated]` -- so the claim is corrected rather than repeated.

Also: state the stdio contract on `run_with_timeout` and drop the three
callers' redundant settings, which it overrode anyway.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* Merge remote-tracking branch 'origin/main' into fix/rocmai-448-drain-child-pipes

Two conflicts, one of them a whole-file collision.

crates/rocm-core/src/process.rs: ROCm#599 extracted a module of the same
name out of lib.rs, so main and this branch each created a different
file at that path. Resolved as a union -- the two have zero name
overlap, main's being spawn/terminate primitives and `FileLock`, this
branch's being `run_with_timeout` and its `Capture` machinery, which is
exactly the module main just created for this kind of thing. Imports
merged (main's narrower `use std::process::Command` dropped for this
branch's wider one; `anyhow::bail` widened to unconditional and the
windows-only duplicate removed), and both test modules joined -- 17
`process::` tests now, main's three `FileLock` cases included.

docs/architecture.md: main added net.rs/paths.rs/process.rs bullets.
Kept them, folding `run_with_timeout` into the process.rs entry. The
previous merge had reinstated the claim that this is "the one bounded
child-process execution helper the apps share", which 6533327 removed on
purpose; rominf caught that, and it is not reinstated here. The bullet
now says plainly that other capture helpers still join their readers
unbounded.

Also fixes jussielo-amd's blocking finding: the wait loop's two `?`
early returns -- `try_wait` and the post-kill `wait` -- returned before
`take()` was reached, so `abandoned` was never set and each reader
thread kept appending to a buffer nobody would ever read, contradicting
`Capture`'s own documented bound. Both paths now abandon first.

Verified on Linux: clippy --workspace --all-targets -D warnings exit 0;
cargo test --workspace --all-targets 32 groups, the only failure being
the known `animated_spinner_keeps_ticking` load flake from ROCm#421, which
passes 3/3 in isolation.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(core): abandon both readers when the wait loop itself fails

Addresses the review on ROCm#526.

`run_with_timeout` starts a reader thread per pipe before it waits, and
`Capture::take()` is the only thing that marks a reader abandoned -- which is
what stops it appending to a buffer nobody will read again once a grandchild
holds the write end open. The wait loop's two error paths, a failed `try_wait`
and a failed post-kill `wait`, returned straight out of the function before
either `take()` was reached, so both readers kept accumulating for as long as
the grandchild lived. In `rocmd`, which probes amd-smi every 60s, that is
unbounded growth, and it contradicted the bound `Capture`'s own doc comment
promises.

The wait loop moves into `wait_bounded`, so the two `?`s become one `Err` arm
in `run_captured` that abandons both captures before propagating. One error
path instead of two is the point: a third `?` added to the loop later cannot
reintroduce the leak. `wait_bounded` is generic over a new private `Waitable`
trait purely so the two failures are reachable from a test -- provoking either
from a real child needs the kernel to refuse to reap a process. Both new tests
fail against the previous shape, leaking ~434KB per stream.

The read-error branch now checks `abandoned` under `buffer`'s lock, the same
one `take()` marks it under, closing the window where a genuine read error
could be stored into a failure slot `take()` had just drained.

`assemble_output` builds its `Output` first and quotes through
`command_failure_detail`, the picker ROCm#584 centralised, instead of repeating the
stderr/stdout/"no output" chain.

`docs/architecture.md` names the four sibling helpers that still join their
readers unbounded and links ROCm#623, and closes the aside in the `common.rs`
bullet. The stub payload comments say ~203KiB (measured: 207,691 bytes, ~3.2
pipe buffers, ~52x the note ceiling) rather than ~150KiB, `write_stub_payload`
says three scenarios, and examine-22's Then step says it checks a bounded note.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(core): silence use_self on the Waitable forwarding impl

clippy::use_self wants `Self::try_wait(self)` inside
`impl Waitable for Child`. `Self` is `Child` there, so that spelling
reads as the trait method being defined rather than the inherent one it
forwards to. Allowed locally with that reason rather than taking a
rename that makes the forwarding harder to follow.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(e2e): assert examine-21/22 notes still quote amd-smi's output

Both steps claimed, in their own comment, that their checks "pull against
each other: dropping the child's output entirely would satisfy the size
check while losing the diagnostic". They did not. The only assertions were
`contains("...[truncated]")` and a 4000-character ceiling, and
`truncate_to_chars(output, 0)` returns exactly `"...[truncated]"` -- so a
quote bound of zero satisfied every assertion in both steps and left a note
with none of amd-smi's output in it. The unit test
`assert_timeout_quote_is_bounded_and_useful` does pin this through
`FLOOD_CANARY`, but the scenarios claimed the end-to-end property and could
not fail on it.

Both steps now also assert the note contains a field value from the first
GPU in the stub payload, which falls inside the quoted head. The literal is
shared with the payload as `STUB_MARKET_NAME` so the assertion cannot drift
away from what the stub actually prints.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(e2e): make the quote assertion mean the head, not just non-empty

Every one of the 1200 stub GPUs carried the same market_name, and each
object is short enough that any 2000-character window holds several
copies. `note.contains(STUB_MARKET_NAME)` therefore proved the quote was
non-empty; a quote that kept the tail, or any middle window, passed it
identically. The doc comment and both panic messages said "head".

Give GPU 0 its own name and assert that. The comment was the side that
was wrong, but the weaker reading is not the one worth keeping: the
quote is built by `truncate_to_chars`, which takes the leading
characters, so a truncation that dropped the head is a real regression
this scenario should catch and could not.

Also drops two stale claims from that comment: the first GPU object is
not ~150 characters, and the quoted text is stderr when stderr is
non-empty and stdout otherwise, not "what the child printed".

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(process): kill the child when try_wait fails, and write the amd-smi stub without the ETXTBSY race

`wait_bounded`'s poll error was the one exit from the wait loop that left
the child running: the `?` propagated with neither `kill()` nor `wait()`,
and `run_captured` only abandons the two captures on that path, so the
child was dropped unreaped. Kill before propagating, matching the timeout
branch's best-effort `let _ =` style.

The existing abandon test only watched the readers, so it passed either
way. `FailingChild` now records `kill`, and the shared helper asserts it
for both wait-loop failure paths.

`write_amd_smi_stub` used `std::fs::write` + `set_permissions`, which
reintroduces the fork race `write_fake_executable` exists to close
(ROCm#649). The new scenarios are `@requires-os:linux` with no
`@requires-gpu`, so they run up to 64-wide on the no-GPU lane, where a
leaked write fd in a sibling thread's child turns the stub's first exec
into an intermittent `Text file busy`. Use the helper, as every other
`*_steps.rs` in that directory already does.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(process): say what the module now does, and stop two comments overclaiming

Review follow-up on the three non-blocking items. Comments only; no
behaviour change, and the diff touches no executable line.

- The module doc still described spawn/terminate primitives and
  `FileLock` only. `run_with_timeout` and its supporting cast are the
  bulk of the file now, and `docs/architecture.md` already names it, so
  the two disagreed about what this module is for. Name it, say what it
  guarantees, and point at ROCm#623 for the helpers not yet converged onto
  it.

- `DRAIN_GRACE` claimed a cleanly-exited child "has at most a buffer or
  two left to collect", on the grounds that the readers run throughout.
  They are *started* before the wait loop but not necessarily scheduled:
  `wait_bounded` polls `try_wait` once with no sleep in front of it, so a
  fast enough child can be observed exited before either reader thread
  has run an instruction. Not a logic bug — 200ms absorbs it with room to
  spare — but the comment claimed more than the code structurally
  provides. Say that the grace covers reader start-up too.

- The `abandoned` guard in the reader's read-error branch is kept, with
  its comment corrected to the case it actually carries. After `take()`
  has returned the guard is only tidy: `take()`/`abandon()` run once per
  `Capture`, so a failure stored later does sit unread. The window it
  genuinely closes is *inside* `take()`, which marks `abandoned` and
  empties the buffer under `buffer`'s lock, releases it, then drains the
  failure slot under a different lock. A read error landing in that gap
  would come back attached to bytes already handed to the caller, and
  `assemble_output` turns any failure on a non-timed-out run into a hard
  "failed to read the output of ..." — a cleanly finished command
  reported as a read failure. So it is not inert, and the lock-before-
  check ordering from the earlier TOCTOU fix is what closes it; both
  stay.

Refs: ROCMAI-448
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(process): pin the empty stdin the helper promises its children

`run_with_timeout` took the null-stdin setting over from its three
callers and its doc now tells them not to set it, but nothing failed
when that line was removed: 19 process tests passed against a mutant
without it. A child that reads stdin would then inherit the daemon's,
wait for input that never comes, and be killed at the deadline.

The new case hands `run_with_timeout` a caller-set stdin that stays
open for the whole call, so the child reaches EOF only if the helper
really does replace it; otherwise the call comes back as a timeout.
Added for cmd as well, since both `rocmd` callers run on Windows.

Also three comments that said more or less than the code does: the
`{error:#}` note gains its causes on the spawn failure too, not only
on the timeout; the monitor branch is pinned by `examine-26` rather
than the `examine-25` it inherited from the branch above; and the
quote bound covers this helper's own message, not every persisted
quote of child output. The `Capture` doc now says that what it bounds
is the memory, not the number of parked reader threads.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(process): say what the stdin case cannot tell apart

The new case pins `null` against "no setting at all", which is the
regression that can happen. It cannot pin `null` against `inherit`:
that needs the test process's own fd 0 to be something other than
EOF, and arranging it is a `dup2` racing every other test in the
binary. An `inherit` mutant was run and passes, so the limit is
written down rather than left for the next reader to rediscover.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs: correct the caller count, the ROCm#623 attribution and the spawn note

Four comment defects, no behaviour change.

- The stdin test helper said "all three callers"; `run_with_timeout` has
  four call sites (`common.rs:134`, `common.rs:227`, `sandbox.rs:121`,
  `main.rs:8314`), which is what the `MAX_QUOTED_OUTPUT_CHARS` doc already
  says.
- `MAX_QUOTED_OUTPUT_CHARS` claimed ROCm#623 tracks both unbounded quotes. It
  lists only the `capture_amd_smi_json` one; the daemon's
  `parse_sandbox_child_output` is named nowhere in it, so the comment is
  narrowed to what the issue actually records rather than editing the issue.
- The `{error:#}` note quoted Linux's spawn error text for a probe `rocmd`
  also runs on Windows, called every no-GPU host an amd-smi-less host
  (`resolve_amd_smi_binary` falls back to a bare `amd-smi` on PATH, so such
  a host fails at a non-zero exit instead), undercounted the failures that
  gain causes this way, and let "Pinned by examine-25" trail a claim
  examine-25 does not assert. Rewritten to pin only the timeout claim.
- `WindowsFixture`'s doc still described a payload that only `flooding`
  writes after the `script`/`flooding` split.

Also records why the `inherit` mutant is left documented rather than caught
by a self-re-execing test.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

---------

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

E2E coverage: assert download progress spinner renders under a PTY

4 participants