Repository navigation
fix: amd-smi timeout kill, fix.rs exit constants, thermal dedupe window, ETXTBSY test stubs, dead dash-tui helpers - #842
Conversation
AmdSmiCollector::run spawned amd-smi with output() but without kill_on_drop. Both callers (the version probe at detection and run_json) await it under a timeout, and a timeout that fires only drops the future, so the child kept running after the caller gave up on it. Reproduced with a fake amd-smi that records its pid and then execs sleep 60: after a 2 second limit expired, the sleeper was still alive five seconds later. The command is now built with kill_on_drop(true). To let a test shorten the limits, run_json now delegates to a new run_json_within that takes the limit, and the version probe moved out of detect_with_binary_inner into probe_version(limit). Behaviour is unchanged: they are called with the same RUN_TIMEOUT and DETECT_TIMEOUT. Two regression tests (run_json and detection paths) assert the recorded child pid is no longer running after the timeout; a zombie awaiting reaping counts as gone. They fail without the fix and pass with it, and fail again with kill_on_drop(false). They read /proc, so they are gated to Linux; the fix itself is a plain tokio setting that applies on every platform. rocm-core is added as a dev-dependency of rocm-dash-collectors only, for write_fake_executable (the same helper other crates use to avoid ETXTBSY). Dev-dependencies are outside the crate-edge allowlist, so the collectors still have no normal dependency on rocm-core. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The runners in fix.rs (run_render_group, the run_unset_override_*,
run_path_export_* and run_hip_visible_devices_* families, and the
helpers behind them) returned bare 0, 3, 4 and 5, while apply and
act_on use exit::OK, exit::NOT_APPLICABLE, exit::FAILED and
exit::DECLINED. The exit module says those names exist so the published
manifest and the code that produces the codes cannot disagree, and the
runner region had none.
The values were already right, so this changes no exit code: 36
`return N;`, 6 tail expressions and the one `if declined { 5 } else
{ 0 }` now name their constant (0 is OK, 3 NOT_APPLICABLE, 4 FAILED,
5 DECLINED). `rc != 0` tests on a child process's own status are not
runner results and are untouched. One doc comment that said
`return 5` now says `return exit::DECLINED`.
fix_runners_return_named_exit_constants_not_integers reads fix.rs from
`pub fn apply` to the test module and fails on any `return N;`, bare
integer tail expression, one-line `{ N }` branch or `=> N,` arm outside
comments. Before the change it reported all 42 literals; after it is
empty, and putting `return 4;` back into run_render_group makes it fail
again. A second test checks the lint itself against samples of each
shape and against named constants, comments and unrelated integers, so
the first cannot go quiet because its patterns stopped matching.
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The thermal watcher decided whether a stop request was already waiting by looking at the 100 most recent proposals. A pending proposal is never dropped from the store, so one that had more than 100 newer proposals after it was missed and a second stop request was queued for the same server. The restart watcher's equivalent check already reads the whole store for exactly this reason. Reproduced first: one pressure event queues a stop proposal for a server, 101 proposals for other servers follow, a second pressure event for the same server is handled. Before the change the store held two pending stop proposals for it; after, one, and the event log records stop_proposal_already_pending. Both checks now share pending_proposal_exists, which reads the whole store, so the two cannot drift apart again; the doc that explained the restart choice moved onto it. The matching rule is unchanged: pending status, the watcher id, the service id, and either the queued action or the tool name. The new test fails without the change. Swapping the helper back to a 100-proposal window also turns the existing restart-side test red. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
missing_report_check_flags_only_lanes_that_ran_without_a_report failed once in a full workspace test run with "gh: /bin/sh: bad interpreter: Text file busy" and passed alone. run_report_check wrote the stub gh with std::fs::write and then set_permissions. A child that another test thread spawned while the stub was open for writing keeps that descriptor until its own exec, and Linux will not execute a file anything still has open for writing, so the stub failed and the check reported "could not list this run's artifacts". Syncing before the chmod does not close it; Linux has no close-on-fork. The stub is now planted with rocm_core::fake_executable:: write_fake_executable, which this file already uses for the preflight stubs. The planting moved into write_gh_stub so a test can exercise it. Reproduced two ways before the change. Running the whole check from 48 threads, 20 times each, hit the same error in 2 of 3 runs (2 and 4 of 960 checks), but takes 15 seconds, so that is not the test. The test gh_stub_runs_first_time_while_other_threads_spawn plants write_gh_stub's stub from 16 threads and runs it at once: before the change 39 to 98 of 640 first runs failed with "Text file busy (os error 26)" in each of several runs; after it, none in several runs. Putting std::fs::write plus set_permissions back into write_gh_stub makes it fail again. The jq-or-skip gate of the existing test was not touched. The check script itself and the other scripts this file writes are run through `bash <path>`, which only reads them, so they cannot hit the race. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Follow-up to the report-check fix, from the sweep for other test code that writes a script with std::fs::write, chmods it and then executes it. The same race applies to each: a child spawned by another test thread while the script is open for writing keeps the descriptor until its own exec, and the script then fails with "Text file busy". Read and converted, all test-only, to write_fake_executable: - the four fake python scripts in the comfyui reinstall tests (three inline, one in plant_installable_release); - the dpkg-query stub in previous_driver_warning_fires_only_when_amdgpu_dkms_is_installed; - fake_uv_for_staging_index in the vllm install tests; - stub_bin_dir in the diagnose tests. The now-unused PermissionsExt imports went with them. No assertion changed, and the affected comfyui, driver_install, vllm install and diagnose tests pass. Read and left alone, because they are not a script written and then run from this process: the dummy plugin file in the examine report test (never executed); the script inside the tarball that fake_uv_archive builds (extracted by the code under test); the directory-mode changes in remote/provision.rs, lemonade runtime_dir.rs, archive_props.rs and automation.rs; and the production chmods in uv.rs, openmpi.rs and lemonade archive_extract.rs. These sites were not seen failing: the race is probabilistic, so there is no failing-before run for them as there is for the report check's stub. This commit is hardening and can be dropped without affecting the fix before it. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Narrowing the ui module to pub(crate) shows these items are reached only from tests. Each was checked by grepping the whole workspace for a user; none has one, so each is deleted rather than wired in. - dock::triptych: a second copy of the wide-layout split. The draw path uses wide_triptych in ui/mod.rs (different minimum centre width and rail offsets); this one had no caller. - chat::transcript_lines: a one-line wrapper over transcript_lines_from_turns. The draw path already calls the latter; the three tests that called the wrapper now call it directly, with the same assertions. The doc comment moved onto the function that remains. - hardware::gpu_visible_count and ROWS_ABOVE_GPUS, which only it used. Its doc said AppState called it to keep gpu_scroll in sync, but no caller exists and gpu_scroll is never advanced; the effective scroll is computed from the selection by scroll_to_show at draw time. It had no test either. Other edits this needs: - the test is_wide_and_triptych_extents lost its triptych half; the is_wide assertions are unchanged and it is renamed is_wide_extents; - a comment in hardware.rs saying gpu_visible_count "stays unit-tested" and is reused by Observe's GPU panel (neither was true) is removed with the function. No test other than that one changed. Coverage: the deletions take out about 13 covered and 8 uncovered lines from a crate of 24.5k, which moves its line percentage up, not down, so the committed floor should hold. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🟡 Changes recommended
Thermal-stop deduplication lacks the required user-visible Gherkin regression scenario verifying that a stop older than 100 newer proposals is not duplicated when the watcher runs through the daemon.
1 open issue
What changed in this PR
This follow-up to the modularization work fixes subprocess cleanup and thermal-proposal deduplication, standardizes exit-code returns, and removes unused dashboard helpers.
Changes:
- Kill timed-out
amd-smichildren and detect pending thermal-stop proposals beyond the latest 100 entries, with regression tests. - Replace literal exit codes with named constants and reuse the shared executable-fixture helper to avoid “Text file busy” failures.
- Remove unused dashboard layout, transcript, and GPU helpers while retaining tests for active behavior.
Verification here was limited to source inspection; tests and builds were not run.
| File | Description |
|---|---|
xtask/src/workflow_contract.rs |
Reuse safe executable fixtures and add concurrent stub-execution coverage. |
engines/vllm/src/install.rs |
Use the shared helper for the fake uv executable. |
crates/rocm-dash-tui/src/ui/tabs/hardware.rs |
Remove the unused GPU visibility helper and stale comments. |
crates/rocm-dash-tui/src/ui/tabs/chat.rs |
Remove an unused transcript wrapper; test the underlying mapper directly. |
crates/rocm-dash-tui/src/ui/dock.rs |
Remove unused layout code and retain width-threshold assertions. |
crates/rocm-dash-collectors/src/amd_smi.rs |
Kill children on timeout and add Linux process-lifecycle tests. |
crates/rocm-dash-collectors/Cargo.toml |
Add the test-only dependency for executable fixtures. |
crates/rocm-core/src/fix.rs |
Use named exit constants and add a source-level regression check. |
crates/rocm-core/src/diagnose.rs |
Create diagnostic stubs through the shared helper. |
Cargo.lock |
Record the collectors’ new test dependency. |
apps/rocmd/src/watchers.rs |
Share full-store proposal checks and test older pending-stop deduplication. |
apps/rocm/src/driver_install.rs |
Create the package-query stub through the shared helper. |
apps/rocm/src/comfyui.rs |
Use the shared helper for fake Python executables. |
Review effort: Balanced
| && (proposal.action == "queue_stop_server_proposal" | ||
| || proposal.tool.as_deref() == Some("stop_server")) | ||
| })) | ||
| pending_proposal_exists( |
There was a problem hiding this comment.
Added automations-12 in eb6fa12 (tightened in eb1687d). It starts a real rocmd run --local-webhook-port 0, posts a thermal alert for a ready managed server to its loopback webhook, queues one more pending proposal for other servers than REVIEWED_PROPOSALS_RETAINED, posts the same alert again, and asserts that the daemon records stop_proposal_already_pending and that exactly one stop request for that service is still pending. With the 100-proposal read put back it fails with "the daemon queued another stop request … 2 are now waiting".
Two differences from the wording above. The newer proposals are seeded straight into the store rather than produced by the daemon, and the final check reads the proposal store, because rocm automations lists only the five most recent proposals, so the older stop request would not appear there. The scenario is Linux only (@requires-os:linux) since the managed server's recorded process is a disposable sleep.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · 4a352f4
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change.
No ticket named in the PR body; it describes itself as "the first batch" of a post-modularization-audit follow-up fixing six independent, atomically-committed defects (amd-smi timeout kill, fix.rs exit constants, thermal dedupe window, an ETXTBSY-flaky xtask test plus its sibling sites, and dead dash-TUI helpers).
Blocking
- The new exit-code lint's regex has real detection gaps, and both the in-code comment and the PR description overclaim what it guarantees —
crates/rocm-core/src/fix.rs(bare_exit_code_literals, and thefix_runners_return_named_exit_constants_not_integerstest built on it)
The regex\breturn\s+\d+\s*;|^\s*\d+\s*$|\{\s*\d+\s*\}|=>\s*\d+\s*,?\s*$does not match a bare exit literal written as a suffixed integer (return 4i32;,return 4_i32;), a hex/octal/binary literal (return 0x4;), or a bare integer inside a single-line match expression (match x { Ok(_) => 0, Err(_) => 4 };— the match-arm alternative is anchored to end-of-line, and the trailing};on the same line breaks it). Traced each of these against the regex directly: none of the four alternatives matches any of these forms. Today's migration is genuinely complete (the regex correctly flags nothing in the current runner code), but the doc comment directly above it states "every runner reports its result throughexit::, so the published manifest and the code that produces the codes cannot come to disagree," and the PR description repeats this as "a source-level test fails on any integer literal returned from a runner" — both guarantees this text-pattern check does not actually provide against a future regression written in one of the missed shapes. A check whose documentation claims a branch it cannot fail on is worse than no check, because it tells the next reviewer the gap is already covered.
Confidence 85 · logic · Fix: broaden the regex for suffixed/hex/octal/binary literals and drop (or rework) the end-of-line anchor on the match-arm alternative — or replace the text scan with a token-level check (e.g. viasyn) — and soften the doc comment/PR text to match what the lint actually covers today.
Non-blocking
None
Decisions for the author
None
Positive signals
write_fake_executable's redesign (writing bytes via a short-livedsh -c 'exec cat > path'child instead of holding an in-process write fd open) correctly closes the ETXTBSY race at its root cause, and all five migrated call sites (comfyui.rs,driver_install.rs,diagnose.rs,engines/vllm/src/install.rs,xtask/src/workflow_contract.rs) now share exactly one implementation of the pattern, with no remaining unmigrated write-then-chmod-then-exec site found in a repo-wide sweep.pending_proposal_existsgeneralizes the stop/restart duplicate-guard by parameterizing the watcher id/action/tool rather than keeping two copies of the whole-store scan, so the two paths can no longer drift back apart the way the bug happened the first time.- The amd-smi regression tests cover both of
run()'s only two call sites (detection andrun_json), leaving no branch of the newkill_on_droplogic untested; confirmed via a repo-wide sweep that this was the onlytokio::process::Commandawaited under atimeoutwithoutkill_on_drop(the one other timeout+subprocess-shaped site,rocm-dash-collectors/src/docker.rs, wraps a Docker client library call, not a spawned child, so it was correctly out of scope).
Deployment notes
None — internal bug fixes and test-only changes; no schema/migration/breaking API change. The new rocm-core dev-dependency on rocm-dash-collectors does not create a crate-layering cycle (verified against cargo xtask check-crate-edges's own dev-dependency exemption and against rocm-core's own dependency list).
What this covered
Full diff origin/main...pr/842 (merge-base against the current origin/main tip, isolating exactly this PR's 6 commits — 13 files, +420/-151, matching the PR API's reported stats) at head 4a352f4c7a1ceaa1f6e124cc0556128f7331622f. Every changed file read completely, plus the surrounding code needed to judge each change in place: the exit module and its constants, every AmdSmiCollector caller, xtask/src/crate_edges.rs's dev-dependency exemption logic, rocm-core/src/fake_executable.rs's full implementation and doc comments, wide_triptych/AppState's GPU-scroll clamping, rocm-core/src/automation.rs's proposal-store loaders, and repeated repo-wide greps for every removed/changed identifier and for the from_mode(0o755) pattern. Checked against this repo's root AGENTS.md (subprocess/FFI checklist, dead-code rule, test-regression requirements) and against git log/git blame on every touched file (no incident history found beyond what each commit already documents). CI on this commit: all required checks green (clippy, ratcheted dash-crate coverage, affected-crate tests, Linux/Windows build-and-test, prek, commit-signatures, license headers, lychee); a few hardware-gated nightly/release E2E lanes were still queued, not failed.
Did not run: the Rust test suite or cargo xtask coverage/check-crate-edges directly (no cargo/rustc available in this review environment) — findings above were verified by tracing code and regex behavior directly rather than by execution, which caps evidentiary confidence at 85 rather than 100. The "prior changes" source-reading pass did not run: there is no mechanism available here to see earlier review rounds on these files without reading this PR's own discussion, which this review deliberately stays blind to by design. Not reconciled against any existing discussion on this PR.
🤖 Generated with Claude Code
juhovainio
left a comment
There was a problem hiding this comment.
I read through the full diff and found nothing that breaks behavior: kill_on_drop(true) is on the only Command in AmdSmiCollector::run, and the merged pending_proposal_exists keeps both old matching rules. The regression tests assert the child pid is actually dead, which is the right check.
I'm requesting changes on the test-hardening points below, which would make the new guards fail loudly instead of silently:
- The exit-code lint in
fix.rsonly scans text after the firstpub fn apply(, so a runner defined above it would go unchecked (inline comment). The regex also missesreturn 3without a semicolon andlet rc = 4;. - The watchers test hard-codes the old 100-proposal window via its 101 filler proposals (inline comment).
gh_stub_runs_first_time_while_other_threads_spawnis probabilistic and exercises the shared helper rather than the xtask call site; a direct unit test of the helper would be the better layer.- The amd-smi tests are Linux-only, so the Windows
kill_on_droppath is untested. The PR text discloses this, so I'm only noting it.
CI: two Strix Halo Ubuntu nightly e2e lanes (current, ~45 min each) are red and the W7900 lane was still pending when I looked. The diff touches no e2e code, so they look like hardware/timeout flakes, but I couldn't pull the logs; worth confirming the same lanes fail on main.
| .split_once("#[cfg(test)]\nmod tests {") | ||
| .expect("fix.rs has a test module"); | ||
| let (_, runners) = production | ||
| .split_once("\npub fn apply(") |
There was a problem hiding this comment.
The lint only scans text after the first \npub fn apply(, so a runner defined above apply isn't checked. Consider asserting the scanned slice contains a known runner (e.g. run_render_group) so a reorder fails loudly instead of silently shrinking coverage.
There was a problem hiding this comment.
The lint no longer slices the text at pub fn apply(. As of 19fa2dc and eb1687d it parses all of fix.rs with syn and inspects every free or impl function returning i32, and the real-file test now asserts that all sixteen of those runners were inspected, so a reorder or a rename fails loudly instead of shrinking the scan. The shapes you listed (return 3 without a semicolon, let rc = 4; then rc) are covered, as are suffixed and hex literals and one-line match arms, each with a case in the lint's own test; locals assigned from an if or match with a literal branch are followed too. I broke a real runner with return 4_i32; and the test went red. The doc comment now lists what the scan still cannot see (trait default methods, nested functions, aliased return types, casts).
| // no longer among the 100 most recent. A pending proposal is never | ||
| // dropped from the store, so it is still waiting for review. | ||
| for index in 0..101 { | ||
| queue_proposal_with_arguments( |
There was a problem hiding this comment.
The 101 filler count encodes the old 100-proposal window. A comment or shared const tying it to the real window would keep this test meaningful if the window changes.
There was a problem hiding this comment.
Both dedupe tests (this one and the server-recover sibling) now queue REVIEWED_PROPOSALS_RETAINED + 1 proposals instead of a hard-coded 101, in a50410a, with the comment saying what that establishes: the pending request sits beyond a recent-window read the size of the reviewed-retention cap. The filler proposals are pending, so compaction never drops them; the claim is that size, not "any window". With the whole-store read swapped for a 100-proposal one, both tests fail.
The lint matched a regex against the text after `pub fn apply(`, so a suffixed (`4i32`), hex or one-line-match literal, a `return 3` with no semicolon, a literal-valued local that is then returned, and any runner defined above `apply` all slipped past it, while the comment above it claimed none could. It now parses fix.rs with syn and inspects every function returning `i32`: a literal counts when it is a `return` operand, the tail expression, an arm of a tail `if` or `match`, or a local set to a literal and returned by name. The real-file test also asserts that known runners were inspected, so a reorder or rename fails loudly instead of shrinking the scan. The doc comment now says what the check cannot see. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
… bound The thermal-stop and server-recover dedupe tests queued 101 newer proposals, a number that only meant something while the code read the 100 most recent. They now queue one more than REVIEWED_PROPOSALS_RETAINED, so the pending request sits outside any recent window a reader could take and the tests keep guarding the whole-store read if the bound changes. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
gh_stub_runs_first_time_while_other_threads_spawn ran the shared write_fake_executable helper under the same 16-thread race that the helper's own a_fake_runs_first_time_while_other_threads_spawn already runs, so it was a second probabilistic copy of that test at the wrong layer. The helper's test carries the race; the xtask report-check tests still run the planted stub through write_gh_stub. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Suppressing a duplicate stop request changes the review list `rocm automations` shows, so the unit test on the handler is not enough on its own. automations-11 starts the real `rocmd run --local-webhook-port 0`, posts the same thermal alert to its loopback webhook twice with more proposals than the store keeps of reviewed ones queued in between, and checks the daemon notes the request is already waiting and leaves one stop request pending. It fails with two pending stop requests when the check goes back to reading only the 100 most recent proposals. docs/automations.md states the behaviour; docs/testing.md names the unit and e2e coverage. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The lint now follows a local assigned from an `if` or `match` with a literal branch, treats any `cfg` that mentions `test` as test code, drops an arm `syn` never produces, and says in its doc comment what it still cannot see (trait default methods, nested functions, aliased return types, casts). Each branch has a case in the lint's own test, and the guard that the scan reaches the runners names all sixteen. automations-11 plants the managed server record as plain JSON instead of through the record type, as the harness README asks, and the steps module header no longer says the file never starts a daemon. The wording about how far outside a recent-window read the filler proposals sit is narrowed to what the tests establish: beyond a window the size of the reviewed retention cap. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The e2e harness lint (clippy::future_not_send, run by the pre-push hook on `--test e2e`) rejects a step future that keeps &E2eWorld across an await. The thermal-stop steps now take an owned AppPaths and poll the stores through it. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
|
Pushed six commits on top of the reviewed head that apply the review feedback so far; the PR description is updated to match.
The |
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · 5964ba4
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change.
No ticket named; the PR describes itself as the first batch of a post-modularization-audit follow-up fixing six independently-committed correctness/dead-code items (amd-smi timeout kill, fix.rs exit constants plus a syn-based regression lint, thermal-stop dedupe window, an ETXTBSY-flaky xtask test plus four sibling sites, and dead dash-TUI helpers), plus review-feedback commits (lint rewrite from regex to syn, test hardening, a new e2e scenario). The description's file-by-file account matches the actual diff; no scope mismatch found.
Blocking
-
A failing CI check at this head commit — GitHub Actions job "E2E tests (Strix Halo, Windows, current)" (workflow "E2E self-hosted"), run 41641403144
The job log shows an unexpected regression:Reconciliation: 3 xfail (failed as expected), 0 XPASS (0 flaky, 0 stale), 1 unexpected failure(s). FAIL: 'runtime-resolve-canonical-release' was expected to pass on this host but FAILED — a regression.None of this PR's 19 changed files touch the "TheRock ROCm 10 install layout" pip-index resolution path this scenario exercises, so this may be a pre-existing flake rather than something this change caused — but it is a real, completed failure on this exact commit and must be resolved or positively confirmed as unrelated/pre-existing (e.g. also failing onmain) before merge.
Confidence 100 · logic · Fix: re-run the lane; if it also fails onmainat a similar commit, document that and treat as a tracked pre-existing flake rather than blocking this PR; otherwise bisect the regression. -
is_cfg_test's word-matching misclassifiescfg(not(test))as test-only, silently exempting such a runner from the new exit-code regression scan —crates/rocm-core/src/fix.rs:4971-4981fn is_cfg_test(attrs: &[syn::Attribute]) -> bool { attrs.iter().any(|attr| { attr.path().is_ident("cfg") && attr.meta.require_list().is_ok_and(|list| { list.tokens .to_string() .split(|c: char| !c.is_alphanumeric() && c != '_') .any(|word| word == "test") }) }) }
This checks only whether the token string contains "test" as a split word, not whether the cfg predicate actually is
test. Verified directly: for#[cfg(not(test))]— which gates code to non-test builds, i.e. production code —list.tokens.to_string()is"not (test)", and splitting on non-alphanumeric characters yields["not", "test"];.any(|word| word == "test")istrue, so the function returnstrueand the scanner exempts the item as if it were test-only. The same would happen for anycfgcombination that merely contains "test" as a token, e.g. a hypotheticalcfg(feature = "test-mode"). This is new code added by this PR specifically to replace a previous regex-based check whose documented purpose is precision against exactly this kind of silent gap; the gap is dormant today only because no runner currently carries such an attribute, and it is not among the limitations the lint's own doc comment or the PR description disclose (both list trait default methods, nested functions, aliased return types and casts, but not this).
Confidence 85 · logic · Fix: parse the cfg predicate structurally (e.g., matchlist.tokensagainstsyn::parse2::<syn::Path>equal totest, or explicitly check for and reject/handle a leadingnot(...)) rather than string-splitting for a bare substring match; add a unit test assertingcfg(not(test))is not treated as test-only.
Non-blocking
None
Decisions for the author
- Using
synfor the exit-code lint despitextask's own ASCII-output lint being deliberately hand-rolled to avoid a dependency — tradeoff
The PR body raises this itself. In favor:synhere is test-only, already compiled transitively for derive macros elsewhere in the workspace, and a parse-level check closes literal-spelling/layout gaps a regex demonstrably had (per the prior review round on this PR). Against: it's a precedent-breaking dependency choice relative toxtask's stated preference for hand-rolled lints, and (per the finding above) a parser-based check is not automatically immune to its own logic bugs — it traded one class of gap for a different, narrower one. Worth the author's explicit confirmation that this precedent is intended to diverge fromxtask's convention rather than needing reconciliation with it.
Positive signals
write_fake_executable's redesign (writing bytes via a short-livedsh -c 'exec cat > path'child rather than holding an in-process write fd open) closes the ETXTBSY race at its root cause; all five call sites now share exactly one implementation, and a repo-wide sweep forfrom_mode(0o755)found no un-migrated site left.pending_proposal_existsgenuinely unifies the stop/restart duplicate-guard onto one full-store-scanning helper (confirmedload_automation_proposals, not the windowedload_recent_automation_proposals, is what's called now), and both the unit tests and the newautomations-11e2e scenario key off the realREVIEWED_PROPOSALS_RETAINEDconstant rather than a hard-coded magic number.- The
kill_on_drop(true)fix covers both ofAmdSmiCollector::run's only two call sites, and a repo-wide sweep confirmed no othertokio::process::Commandawaited under atimeoutanywhere in the tree is missing the same guard. - The dash-TUI dead-code removal was independently re-verified as genuinely unreachable (not just unreachable-looking):
dock::triptychhad no caller even before this PR and is a different (not byte-identical) layout from thewide_triptychthat replaces it architecturally, and the portedchat::transcript_linestests carry forward byte-identical assertions. - The exit-code lint's doc comment is otherwise honest about several real limitations (trait default methods, nested functions, aliased return types, casts) rather than overclaiming — the
is_cfg_testgap above is a logic bug the comment doesn't know about, not a known gap the comment hides.
Deployment notes
None — internal bug fixes and test-only changes; no schema/migration/breaking API change. The new rocm-core dev-dependency on rocm-dash-collectors, and syn as a new dev-dependency of rocm-core, do not create a crate-layering cycle (xtask/src/crate_edges.rs exempts dev-dependencies from its allowlist check, and rocm-dash-daemon already has an equivalent dev-dependency on rocm-core as precedent).
What this covered
Full diff 61825309b62d1a1b03d28eb49731ffe10dcf8fb6...5964ba4af026a6c24644ee1f13cebeaafd013b3a (19 files, +1037/-161, matching the PR API's reported stats) at head 5964ba4af026a6c24644ee1f13cebeaafd013b3a. Every changed file read completely by five independent fan-out passes grouped by coupling (amd-smi collector + its Cargo manifests; fix.rs's exit-constant conversion and its full syn-based lint, read in full across 8 chunks; thermal-watcher dedupe plus the new e2e scenario and its docs; the ETXTBSY fake-executable migration across all 5 call sites plus xtask; dead dash-TUI helper removal), each also reading the surrounding code needed to judge the change in place (AmdSmiCollector's callers, rocm-core/src/automation.rs's store loaders, fake_executable.rs's full implementation, xtask/src/crate_edges.rs's dev-dependency exemption, AppState's GPU-scroll clamping, ui/mod.rs's wide_triptych, the e2e harness's scenario wiring) and cross-checking against this repo's root AGENTS.md (the kill_on_drop/subprocess rule, platform-gated-test-coverage rule, dead-code rule). I independently re-verified the surviving is_cfg_test finding myself directly against the source before including it. CI was read live via gh pr view --json statusCheckRollup and the one failing job's log was pulled directly rather than assumed; three hardware-gated lanes (MI300X, Strix Halo Ubuntu release, Strix Halo Ubuntu nightly) were still queued/in-progress at review time and are not otherwise reflected here.
A prior automated review by this same identity exists on this PR at an earlier commit (4a352f4), but the code changed substantially since then — the exit-code lint was rewritten from regex to a syn-based parser (a different implementation, not an edit to the one previously reviewed), a new ~300-line e2e scenario and step-definition set were added, and watchers.rs/xtask were further revised — so the prior review's one open finding no longer describes code that exists, and narrowing was not safe by the method's own test; this is accordingly a full fresh review of the whole current diff rather than a narrowed one.
Did not run: the "prior changes" source-reading pass (no mechanism to see earlier review rounds without reading this PR's own discussion, which this review stays blind to by design). No cargo/rustc toolchain was available in this review environment, so no test or lint was executed directly by this review itself; the two findings above were established by direct source tracing (and, for the CI finding, by reading an already-executed job's actual log rather than assuming its result). Not reconciled against any existing discussion on this PR.
🤖 Generated with Claude Code
juhovainio
left a comment
There was a problem hiding this comment.
I read the whole diff and found nothing blocking. The kill_on_drop fix, the shared pending_proposal_exists helper, the syn-based exit-code lint and the daemon scenario all look sound. This reflects a diff-only read; I did not build or run anything.
Non-blocking notes:
- The amd-smi timeout tests use a 2s limit and require the fake script's pid file to exist, which can flake on a loaded runner (inline comments below).
- The lint's scan_items doesn't skip #[cfg(test)] on impl blocks (inline).
- The dedupe tests size their filler from REVIEWED_PROPOSALS_RETAINED, while the old window was a literal 100, so they only guard the old bug while that cap is at least 100.
- Please double-check that the
cargo test -p rocmd older_than_the_recent_windowfilter in docs/testing.md actually matches the restart test. - Six items in one PR stretches the one-logical-change guideline; fine here since the description is explicit about scope.
CI: the Strix Halo Ubuntu E2E failed at job setup with "No space left on device" (runner infra). The Strix Halo Windows E2E failed in its test step and I couldn't see the cause; the new scenario is Linux-only so it's probably unrelated, but worth a look before merge. The remaining GPU jobs were still pending when I checked.
| }; | ||
|
|
||
| let err = collector | ||
| .run_json_within(Duration::from_secs(2), &["metric", "--json"]) |
There was a problem hiding this comment.
A 2 s limit plus a hard requirement that the fake script already wrote its pid file can fail spuriously on a loaded runner ("never started"). Consider a larger limit, or waiting for the pid file before starting the clock.
There was a problem hiding this comment.
Raised the limit to 10 s in 31e0023, for this test and the detection one, through one named constant with a comment on why. The clock starts at the call, so I could not wait for the pid file before starting it; a limit well above what a loaded runner needs to start a shell was the option that keeps the test deterministic. Both tests still pass and run in about 10 s in parallel.
|
|
||
| assert!( | ||
| collector | ||
| .probe_version(Duration::from_secs(2)) |
There was a problem hiding this comment.
Same concern as above for the detection-timeout test.
There was a problem hiding this comment.
Same change: the detection-timeout test uses the same 10 s constant (31e0023).
| syn::Item::Fn(function) if !is_cfg_test(&function.attrs) => { | ||
| scan_function(&function.sig, &function.block, scan); | ||
| } | ||
| syn::Item::Impl(imp) => { |
There was a problem hiding this comment.
impl blocks aren't checked for #[cfg(test)], unlike fns and mods, so a test-only impl returning a literal i32 would be flagged. Low value, but reusing is_cfg_test(&imp.attrs) here would make it consistent.
There was a problem hiding this comment.
Done in 31e0023: impl blocks now go through is_cfg_test like fns and modules, with a case for a #[cfg(test)] impl in the lint's negative test. While there I also tightened is_cfg_test itself, which matched any cfg mentioning test: it now reads the predicate, so cfg(not(test)) and cfg(any(test, windows)) are scanned as production code and only cfg(test) and cfg(all(test, ..)) are skipped. Both new shapes have a case in the positive test, and the test goes red when not is treated like all.
The exit-code lint decides what is test-only from the cfg predicate rather than from whether it mentions `test`: `cfg(not(test))` and `cfg(any(test, windows))` are production code and are scanned, `cfg(test)` and `cfg(all(test, ..))` are skipped, and `impl` blocks are checked like fns and modules. The two amd-smi timeout tests give the collector 10 s instead of 2 s: the clock starts at the call, so a loaded runner that needs a couple of seconds to start the fake script ended the run before the child existed. The dedupe tests size their filler as the larger of the reviewed-retention cap and the 100 the code once read, so they keep guarding that window if the cap is lowered. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
|
Second round, pushed as 31e0023:
On the Windows self-hosted lane: the one failure was |
The one conflict was apps/rocm/src/driver_install.rs: main moved the driver plan code and its tests into driver_plan.rs, which carried the dpkg-query stub this branch plants with write_fake_executable. The file takes main's version and the stub change is re-applied in driver_plan.rs. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Re-applying the write_fake_executable change in driver_plan.rs after the merge left use std::fs unused in that test module. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The conflict was tests/e2e-cucumber/features/automations.feature: main added automations-11 (the background helper's lifecycle) where this branch added automations-11 (the thermal stop dedupe). Main's keeps the number and this branch's becomes automations-12, with the references in docs/testing.md and the steps module header updated to match. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · 1bf9861
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change.
No ticket named. The PR describes itself as the first batch of a post-modularization-audit follow-up fixing six independently-committed items (amd-smi timeout kill, fix.rs exit constants plus a syn-based regression lint, thermal-stop dedupe window, an ETXTBSY-flaky xtask test plus four sibling sites, dead dash-TUI helpers) plus review-feedback commits. The description's file-by-file account matches the actual diff; its own disclosed lint-coverage caveats match what the code states.
Blocking
-
This PR branch has a real, current merge conflict with
main's tip, including a genuine scenario-ID collision —tests/e2e-cucumber/features/automations.feature,docs/automations.md:35
Confirmed withgit merge-tree $(git merge-base origin/main 1bf9861) origin/main 1bf9861(a real run, not an assumption from GitHub's cachedmergeable_state): the only conflict hunk is inautomations.feature, where this PR's new scenario and a different scenario already merged intomainboth claim the sequential title "automations-11" with different@id:tags (@id:automations-thermal-stop-not-requeued-behind-newer-proposalshere vs.@id:automations-helper-stops-when-no-watcher-is-enabledonmain).docs/automations.mdadded by this PR references "automations-11" by that number, which goes stale the moment the collision is resolved.
Confidence 100 · architectural · Fix: rebase onto currentmain, renumber this PR's new scenario (e.g. to automations-12), and update thedocs/automations.mdreference. -
The new
syn-based exit-code lint has an undisclosed logic gap: a two-hop local-variable alias chain escapes detection —crates/rocm-core/src/fix.rs(yields_literal,FunctionResults::result/visit_local)
yields_literalbuilds a throwawayFunctionResults::default()per call and returns!branches.literals.is_empty(); for aPathexpression (a bare variable reference)FunctionResults::resultonly ever pushes intoreturned_names, neverliterals. So forfn via_chained_locals() -> i32 { let code = 4; // correctly recorded in the real scan's literal_locals let rc = code; // visit_local calls yields_literal(code) -> false (Path isn't a literal) rc // "rc" is never in literal_locals -> not flagged }
traced by hand against the actual code:
codeis correctly tracked inresults.literal_localsby the outer scan, butrc's initializer is itself aPath, andyields_literalnever consults the realliteral_localsset when resolving one — sorcis returned as a bare literal and the lint stays silent. The doc comment onscan_exit_code_literalsexplicitly claims detection of "a local that was set to such a value and is then returned by name" without this one-hop qualifier, and lists only trait default methods, nested functions, aliased return types, casts, helper calls and struct fields as exclusions — a two-hop alias chain is not among them. Per this repo's own rule on checks that overclaim their coverage, this is a regression-prevention lint whose documented guarantee is broader than what it verifiably checks.
Confidence 85 · logic · Fix: have the literal check consult the realliteral_localsaccumulated by the outer scan when resolving aPath(not a fresh, throwawayFunctionResults), and add a test case for a chained/aliased local.
Non-blocking
- Doc comments describing the new amd-smi timeout tests get the comparison backwards —
crates/rocm-dash-collectors/src/amd_smi.rs:112,177-178
Both comments justify parameterizing the timeout "so a test can shorten the limit," but the test'sTEST_LIMIT(10s) is longer thanDETECT_TIMEOUT(5s) and exactly equal toRUN_TIMEOUT(10s) — never shorter. No behavioral impact.
Confidence 85 · mechanical · Fix: reword to "so a test can set an explicit limit."
Decisions for the author
- Using
synfor the exit-code lint despitextask's own ASCII-output lint being hand-rolled to avoid a dependency — tradeoff
The PR body raises this itself. In favor: the dependency is test-only, already compiled transitively elsewhere in the workspace, and a parse-level check closes the literal-spelling/layout gaps a text-pattern check would have. Against: it's a precedent-breaking dependency choice relative toxtask's stated preference, and (per the finding above) a parser-based check is not automatically immune to its own logic bugs. Worth the author's explicit confirmation this precedent is intended. - The lint's reach is scoped to one file, and "runner" is identified purely by
-> i32signature — non-blocking-improvement
scan_exit_code_literalsonly sees text physically insidefix.rsviainclude_str!; a future split of this ~5300-line file would silently blind it to a relocated runner. Separately, "runner" means "any free/impl function returningi32" with no tie to the actual recipe catalog — harmless today (verified the only 16 such functions are the real runners) but a future unrelatedi32-returning helper could false-positive-fail the test. Neither is wrong today; both are worth a comment noting the assumption so a future split doesn't quietly defeat the check.
Positive signals
kill_on_drop(true)on the amd-smi child covers both call sites that await it under atimeout(probe_version,run_json_within), and the new regression tests assert actual child death via/procstate rather than just a returned error — reverting the fix makes them fail.pending_proposal_existsgenuinely unifies the stop/restart duplicate-guard onto one full-store-reading helper, preserves each side's original matching predicate exactly, and both the unit tests and the new e2e scenario tie their filler-proposal counts to the realREVIEWED_PROPOSALS_RETAINEDconstant rather than a hard-coded number.write_fake_executable's redesign (a short-lived externalsh -c 'exec cat > path'writer instead of an in-process fd) closes the ETXTBSY race at its root cause; a repo-wide sweep found no remaining write-then-chmod-then-exec site left unmigrated among the five call sites.- The dash-TUI removals were independently re-verified as genuinely dead rather than taken on faith:
dock::triptychwas never the code path the real renderer uses (a distinct, not byte-identical,wide_triptychis), and the ported tests carry forward identical assertions rather than losing coverage. - The new lint's
cfg(test)/cfg(not(test))/compound-cfg handling is correct and is itself exercised by adversarial fixtures in its own test.
Deployment notes
Internal bug fixes and test-only changes; no schema/migration/breaking API change. The new rocm-core dev-dependency on rocm-dash-collectors, and syn as a new dev-dependency of rocm-core, are both layering-safe (xtask/src/crate_edges.rs exempts dev-dependency edges, confirmed against its own test). This PR cannot merge as-is regardless of the above: see the first Blocking finding — it needs a rebase before it can land.
What this covered
Full diff 08235c55c137f1620f2afeb34ceb8ac7810d5133...1bf98611b73a0730d25dea251069db18a61b04ee (19 files, +1080/-163, matching the PR API's reported stats). Five independent fan-out workers, grouped by coupling, each read their full file group completely plus necessary surrounding context: (A) the amd-smi collector and its Cargo manifests, plus every tokio::process::Command+timeout site repo-wide; (B) fix.rs's full ~5300-line head revision, including every migrated return site and the new lint's complete AST-walking logic; (C) the thermal-watcher dedupe plus the new e2e scenario, docs, and the proposal-store's loader/retention code; (D) the ETXTBSY migration across all 5 call sites plus a repo-wide sweep for any missed from_mode(0o755) site; (E) the dash-TUI dead-code removal, cross-checked against AppState's GPU-scroll clamping and the real renderer's wide_triptych. I independently re-verified the surviving fix.rs finding myself by tracing the exact code paths. Driver-level checks beyond the five groups: a live git merge-tree run against the current origin/main tip (not just GitHub's cached mergeable_state), a scenario-ID cross-check against main's own automations.feature, CI check-runs at this exact head (26 runs, all success/skipped, none failing), and commit-message/structure quality (all descriptive, no noise commits).
No cargo/rustc toolchain was available in this environment; no test, lint or build was executed directly — findings were established by direct source tracing and by reading already-completed CI results, not by running anything new. The "prior changes" source-reading pass did not run: this review stayed blind to this PR's own discussion and any earlier review rounds by design; I confirmed independently via git that the current head is past every previously-reviewed commit, so this is a full review of the whole current diff rather than a narrowed one. One fan-out worker initially failed to launch (a concurrent-subagent limit, not a worker failure) and was respawned successfully; all five groups' full results are reflected above. Not reconciled against any existing discussion on this PR.
🤖 Generated with Claude Code

Why
The closing audit of the modularization work found four real correctness defects, some dead code and some stale test scaffolding that are not pure moves, so the move PRs could not carry them. This change fixes the defects, each with a regression test that fails without the fix, and removes the dead dash-TUI helpers that nothing outside tests calls. This is the first batch of that follow-up work.
What changes
Six commits, one per item, so each can be reviewed, kept or dropped on its own, followed by seven commits that apply review feedback (described after the items):
AmdSmiCollectorawaited the spawnedamd-smiunder atimeoutfor both detection andrun_json, but built the command withoutkill_on_drop(true), so when the timeout fired the future was dropped and the child kept running (AGENTS.md section 6 requires the kill). The command now setskill_on_drop(true). To test it, the limit became a parameter of two small functions (run_json_withinandprobe_version) that production calls with the existingRUN_TIMEOUTandDETECT_TIMEOUT.rocm-coreis added as a dev-dependency ofrocm-dash-collectorsfor the fake-executable helper; normal dependencies are unchanged.fix.rsrunners returned bare integers instead of theexit::constants. The runners returned literal 0, 3, 4 and 5 (42 sites) while theexitmodule exists so the published manifest and the code that produces the codes cannot disagree. They now returnexit::OK,NOT_APPLICABLE,FAILEDandDECLINED, whose values are the same numbers, so behaviour is unchanged. A test that parsesfix.rswithsyn(a new test-only dependency ofrocm-core, already in the lockfile) fails on an integer literal that is a runner's result: areturnoperand, the tail expression, an arm of a tailiformatch, or a local set to such a value and returned. It cannot see trait default methods, nested functions, aliased return types or casts; its doc comment lists them.pending_stop_proposal_existsread a recent window while its restart sibling reads the whole store, so a pending stop proposal older than 100 newer ones was missed and a second one was queued. Both checks now share one helper that reads the whole store; the matching rule is unchanged.xtasktest.missing_report_check_flags_only_lanes_that_ran_without_a_reportsometimes failed with "Text file busy", because the stubghwas written and thenchmoded while a child spawned by another test thread still held the write descriptor. The stub is now planted with the existingwrite_fake_executablehelper. (The dedicated stub test was removed in the follow-up below.)from_mode(0o755)hit: the fake Pythons in the ComfyUI reinstall tests, thedpkg-querystub in the driver-install test, the fakeuvin the vLLM tests andstub_bin_dirin the diagnose tests now use the same helper. These sites were not seen failing, so this commit has no failing-before test; it is separate so it can be dropped without affecting item 4.dock::triptych(a duplicate ofwide_triptychinui/mod.rs, which is what drawing uses),chat::transcript_lines(a one-line wrapper whose tests now calltranscript_lines_from_turnswith the same assertions), andhardware::gpu_visible_countwithROWS_ABOVE_GPUS(documented as used byAppState, but nothing calls them) are removed. One test lost its dead half and was renamedis_wide_extents; itsis_wideassertions are unchanged.Review follow-up commits:
The exit-code lint was rewritten from a regex over text to the
synparse described in item 2, after review found suffixed, hex, no-semicolon and one-line-matchliterals, locals returned by name, and runners aboveapplyall slipped past the regex; a later commit follows locals throughif/matchbranches, treats anycfg(..test..)as test code, and names all sixteen-> i32runners in the guard that the scan reaches them.The two watcher dedupe tests now queue one more proposal than
REVIEWED_PROPOSALS_RETAINEDinstead of a hard-coded 101.gh_stub_runs_first_time_while_other_threads_spawninxtaskwas removed: it ran the shared helper under the same race as the helper's owna_fake_runs_first_time_while_other_threads_spawninrocm-core, so it was a second probabilistic copy at the wrong layer.New end-to-end scenario
automations-12(Linux only, mock lane): it starts a realrocmd run --local-webhook-port 0, posts the same thermal alert twice with more proposals than the store keeps of reviewed ones queued in between, and checks that the daemon records the stop request as already waiting and leaves one pending. With the 100-proposal read put back it fails with "the daemon queued another stop request … 2 are now waiting".docs/automations.mdanddocs/testing.mddescribe it.Second review round:
is_cfg_testreads the cfg predicate (socfg(not(test))is scanned,cfg(test)andcfg(all(test, ..))are skipped) and coversimplblocks; the two amd-smi timeout tests use a 10 s limit so a loaded runner cannot end the run before the fake child starts; the dedupe tests size their filler as the larger of the retention cap and the old 100.Decision for the reviewer: the lint uses
synalthoughxtask's ASCII-output lint is deliberately hand-rolled to avoid a dependency. Here the dependency is test-only and already compiled for the derive macros, and it removes the literal-spelling and layout gaps a regex has.Verification
amd-smi child … was still running after its timeout fired) and with two pending stop proposals instead of one; item 2's lint reports all 42 literals before andleft: ["return 4;"]when one is put back; item 4's stub test (since removed, see above) failed 39 to 98 of 640 first runs with "Text file busy" before and none after, and the helper's own concurrent-spawn test now carries that race.cargo fmt --checkis clean andcargo clippy -D warnings --all-targetsis clean forrocm-dash-collectors,rocm-core,rocmd,xtask,rocm,rocm-engine-vllmandrocm-dash-tui.cargo xtask manifest --checkandcargo xtask check-crate-edgespass.xtasktest described in the follow-up list.rocm-corefix::tests,rocmdandxtaskunit tests, and theautomations-*scenarios (11 of 11, three consecutive runs) against debug builds ofrocmandrocmd.tests/e2e-cucumbersuite and the coverage floors (thextaskfloor loses about 35 covered lines to the removed test and may sit near its tolerance; item 6 removes about 13 covered and 8 uncovered lines from a 24,500-line crate, so its floor should hold; no floor was re-blessed). The amd-smi tests read/proc, so they run on Linux only; the fix itself applies on every platform.Scope
The exit-code values are unchanged and the kill is internal, so items 1 and 2 need no end-to-end scenario. The thermal-stop dedupe (item 3) is user-observable only once more than 100 newer proposals sit behind a pending stop request (a second identical request used to be queued); that is covered by
automations-12, and the unit tests cover the handler. The sibling restart dedupe forserver-recoverhas unit coverage only. The remaining items of the audit follow-up are not in this batch because they touch files that other open PRs are reshaping or need a decision first. The skills model inskills.rs(the pure planning code for the two built-in skills, whose executor no longer exists) and the card-grid helpers used only by the dead card grid are deliberately kept pending a decision on whether to delete or rebuild them.Template checklist: this PR fixes bugs but no tracked xfail rows are involved; no new subcommand or subsystem; no user-facing message is added or changed.