Skip to content

fix(dash): detect amd-smi through WSL DXG (EAI-8450) - #375

Merged
juhovainio merged 9 commits into
mainfrom
fix/wsl-dashboard-amd-smi
Oct 2, 2026
Merged

juhovainio merged 9 commits into
mainfrom
fix/wsl-dashboard-amd-smi

Conversation

@michaelroy-amd

@michaelroy-amd michaelroy-amd commented Sep 9, 2026 •

Copy link
Copy Markdown
Member

Summary

  • accept readable /dev/dxg as the WSL transport for the amd-smi safety preflight
  • preserve the existing /dev/kfd guard and the protection against uninterruptible amd-smi hangs
  • rename KFD-specific test seams and make dashboard warnings transport-neutral
  • add @id:dash-reports-wsl-gpu-telemetry for live WSL dashboard coverage
  • distinguish "amd-smi found nothing despite a reachable GPU" from "GPU device inaccessible" in the dashboard warning, and wrap the GPU warning panel's text instead of clipping it

Tracks ROCM-29846.

Why

The WSL amd-smi backend uses ROCDXG through /dev/dxg; /dev/kfd is correctly absent. The dashboard rejected the collector before invoking the working wrapper, producing zero GPUs and an unknown model.

This checks the transport directly rather than treating every WSL environment as safe: production invokes amd-smi only when /dev/kfd or /dev/dxg can be opened.

A follow-up fix in this same PR closes a related gap surfaced during review: ROCDXG-plumbing-readiness (the ROCm-core WSL verdict has_usable_amd_gpu() relies on) is proof the bridge is present, not proof amd-smi can enumerate a supported GPU model. On unsupported WSL GPUs (e.g. a pre-Strix APU like the Radeon 780M/gfx1103), the preflight now passes as before, but if amd-smi still finds nothing, the dashboard warns about that specific contradiction instead of falling back to a generic, self-contradictory "device inaccessible" message. The GPU warning panel's Paragraph was also missing .wrap(...), clipping this (and any other) long warning text at the panel edge.

The WSL-reachability substitution in the preflight is gated on rocm_core::is_wsl_host() (apps/rocm/src/dash.rs), so bare-metal Linux behaviour is byte-identical to a plain /dev/kfd check — has_usable_amd_gpu() alone is deliberately fail-open on an unprobeable platform, which is the wrong polarity for this hang guard. The gate's body (is_wsl_host && has_usable_gpu) is pinned by a truth table; the call site's binding of is_wsl_host() itself is not pinned by any test, since on a GPU-less host the second conjunct already forces the gate false regardless — see the comment above gpu_reachable_for_preflight_only_substitutes_on_wsl in apps/rocm/src/dash.rs.

Verification

  • cargo test -p rocm-dash-collectors — 107 passed, 2 ignored
  • cargo test -p rocm-dash-daemon — 42 passed (adds a contradiction-message regression test, plus positive/negative integration tests driving amd_smi_gpu_reachable through the real (non-test-bypass) device preflight, per review)
  • cargo test --workspace --all-targets — passing aside from pre-existing, environment-coupled failures unrelated to this change (this dev box has a real /opt/rocm install that rocm-core's path-fix test trips over, and a TCP-port-race in a comfyui status test under full-suite parallelism)
  • cargo clippy --workspace --all-targets — clean
  • cargo fmt --all -- --check
  • The new @id:dash-reports-wsl-gpu-telemetry scenario is tagged @requires-wsl @requires-gpu. It is not exercised by any CI lane today: two lanes run @requires-wsl scenarios — the per-PR e2e-wsl job in .github/workflows/e2e-selfhosted.yml and its nightly mirror e2e-wsl-nightly in .github/workflows/nightly.yml — but both deliberately skip provisioning the WSL ROCm driver bridge, so @requires-gpu always resolves to skip on either. The scenario is real, human-runnable coverage of the fix on a properly provisioned WSL box, but CI green on this PR does not include having run it.

Review status

The prior round's blocking item — a call-site binding test for the WSL gate (runner_options_gates_gpu_reachable_wiring_off_wsl) — turned out to pass vacuously on every host without a real, kernel-visible AMD GPU, since the gate's second conjunct is already false there regardless of the first. The head commit drops that test and instead documents the gap honestly in the comment above the truth-table test: the helper's body is pinned, but the call site's binding of is_wsl_host() is not pinned by any test in this crate. The latest automated review confirms this is a net gain in honesty and a net zero in real coverage, with no new defects introduced and no blocking items remaining.

@michaelroy-amd
michaelroy-amd requested a review from a team as a code owner September 9, 2026 21:23

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 1611f58

Summary

Widens the dash amd-smi pre-flight from /dev/kfd-only to /dev/kfd OR /dev/dxg so GPU telemetry works under WSL, renames the skip-preflight plumbing to match, and adds one unit test plus one @requires-wsl @requires-gpu cucumber scenario. Needs work — the new e2e step cannot pass on any host because it polls for a string the TUI never renders at the size the step sets. Verified: I read the full diff and the surrounding render/driver code, and independently confirmed the blocking finding (dock::gpu_wall is the sole producer of "GPUs · N", called only under dock::is_wide ≥180×45 at ui/mod.rs:83-85, while use_detail_size() sets 120×40 at tui_driver.rs:45-46); revert question — the unit test does discriminate (a KFD-only gpu_device_accessible makes it fail; confirmed by running cargo test -p rocm-dash-collectors --lib amd_smi, 11 passed), and the e2e step is not vacuous-pass but guaranteed-fail, so neither is the "passes either way" defect, though the unit test never covers the production call site's use of the DXG_DEVICE constant; sibling-PR interaction — no new WSL predicate is introduced, rocm_core::is_wsl_host (lib.rs:2357-2374) remains the single decider and is untouched, and this change moves in the opposite direction to the open detection-widening PR (it enables GPU reporting rather than suppressing a catalogue, and a native host inheriting WSL_DISTRO_NAME still passes here via /dev/kfd), so it neither compounds that risk nor the host-mutating driver-install PR; no prompt-injection content found in the diff; I relied on the reported CI (26 pass / 0 fail) for the full suite and Windows/GPU lanes, which is precisely why the blocking issue is invisible — the only lane that runs this scenario (e2e-wsl) is non-blocking per tests/e2e-cucumber/README.md:168-183. Blocking: 1 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

tests/e2e-cucumber/tests/e2e/dash_steps.rs:270-273 — the step's only load-bearing condition can never be true, so this scenario will time out and fail on every run, with or without the production fix. The step calls tui.use_detail_size() (line 264), which resizes the PTY to DETAIL_COLS=120, DETAIL_ROWS=40 (tests/e2e-cucumber/tests/e2e/tui_driver.rs:45-46, 273-290). The string "GPUs · N" is produced at exactly one place, crates/rocm-dash-tui/src/ui/dock.rs:74 inside gpu_wall, and gpu_wall has exactly one call site, crates/rocm-dash-tui/src/ui/mod.rs:85, reached only when dock::is_wide(area.width, area.height) is true — which requires cols >= 180 && rows >= 45 (dock.rs:32-33, 40-42). 120×40 fails both axes, so the wide triptych is never laid out and "GPUs · " never appears on screen. The loop at lines 266-284 therefore always exhausts default_timeout() and panics. This blocks because the scenario is the PR's only user-observable evidence for the fix (AGENTS.md §3 requires it), and it currently proves nothing — worse, it fails for a reason unrelated to the change, which will read as "the WSL fix doesn't work". Fix: add a wide-layout resize helper to tui_driver.rs (there is none today — use_detail_size is the widest, and it is too small) that sets at least dock::WIDE_COLS×dock::WIDE_ROWS, call it instead, and re-verify on the e2e-wsl lane; alternatively assert on a GPU signal the single-column layout actually renders.

Non-blocking

  • tests/e2e-cucumber/tests/e2e/dash_steps.rs:280 — !screen.contains("amd-smi unavailable") is inert: that text is rendered only by draw_gpus in crates/rocm-dash-tui/src/ui/tabs/hardware.rs:227-250, which this scenario never navigates to, and only when snap.gpus.is_empty() — a state the GPU-count condition already excludes. Drop it or navigate to the panel that shows it.
  • tests/e2e-cucumber/tests/e2e/dash_steps.rs:274 — "Node throughput ·" is emitted unconditionally by node_throughput_title (crates/rocm-dash-tui/src/ui/tabs/home.rs:131, and see its own test asserting "Node throughput · Unknown GPU" at home.rs:571), so only the paired !contains("Unknown GPU") carries signal; the bare-prefix check adds nothing.
  • crates/rocm-dash-collectors/src/amd_smi.rs:375-381 — the new test injects tempfile paths into gpu_device_accessible and never touches DXG_DEVICE or the call site at line 75-77; a typo in the constant (/dev/dxgkrnl is a plausible one) or a copy-paste passing KFD_DEVICE twice would ship green. One cheap line — assert_eq!(DXG_DEVICE, "/dev/dxg"), or calling gpu_device_accessible(Path::new(KFD_DEVICE), Path::new(DXG_DEVICE)) — closes it.
  • docs/ci-hardware-testing.md:72-74 — "the one scenario whose premise is a WSL host carries @requires-wsl" is now stale; this PR makes it two.
  • tests/e2e-cucumber/features/dash.feature:127 and dash_steps.rs:282 — the tracker reference sits in the @id and the panic message, inverting the convention one scenario above (which keeps a descriptive @id and puts the reference in a comment); the prefix used here is also the only one of its kind in the repo. Separately worth a line in the PR text: /dev/dxg being openable is weaker than WslSummary::rocdxg_ready (crates/rocm-core/src/lib.rs:2486-2489, which also needs dxcore/librocdxg/ldconfig), so dash can now report a GPU on a host rocm examine calls wsl_rocdxg_missing.

rominf
rominf previously requested changes Sep 15, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read through this one closely, and I want to start with what it gets right, because the hard part — the diagnosis — is correct and well targeted. The pre-flight really is what blanks the WSL dashboard; nothing downstream of it is doing WSL-specific gating. I checked that independently: the dash crates contain no other WSL or KFD gating at all, so this is the right layer for the reported symptom. The rename is thorough (doc comments updated in step, no leftover kfd_preflight/kfd_accessible anywhere), the warning strings were made transport-neutral rather than left to rot, and parameterizing gpu_device_accessible over the two device paths is a sensible testability seam.

That said, I think there are four things that block merge, and one of them is a design/safety question rather than a line fix.

1. The new e2e step can never pass. "GPUs · " has exactly one producer — gpu_wall in crates/rocm-dash-tui/src/ui/dock.rs — reached only behind dock::is_wide, which needs cols >= 180 && rows >= 45. The only resize helper the driver offers, use_detail_size(), sets 120x40 (spawn size is 80x24). So on a fully ROCDXG-ready WSL box the step can only time out and panic "WSL GPU telemetry did not appear" — the fix's only user-facing evidence would report the fix as broken. Details and the unsafe obvious workaround are in the inline comment.

2. The scenario runs on no lane, so the CI coverage the description claims does not exist. The capability gate requires driver_status == wsl_rocdxg_ready, and the e2e-wsl job for this exact head (run 34406697117, job 102651494805) reports wsl_rocdxg_missing / librocdxg: false / has_amd_gpu: false. The run's own platform.json records the scenario as "expected": "skip". Skips are filtered before cucumber counts them, so the lane goes green with the scenario never executed. To be fair to you: the runner state is a pre-existing problem already tracked as #223 — that explains the gap, it just doesn't close it, and AGENTS.md is explicit that a unit test doesn't discharge the e2e requirement.

3. The pre-flight invents a third WSL GPU verdict and widens it into a safety question. The repo already consolidated this into rocdxg_ready() (dxg device AND dxcore AND librocdxg AND ldconfig), specifically so examine and serve stop disagreeing — its doc comment says so verbatim. /dev/kfd || /dev/dxg is a fourth, looser answer, and your own CI box is the proof case: dxg present, librocdxg absent, so this pre-flight passes while examine prints wsl_rocdxg_missing and serve refuses. The higher-severity angle is in the inline comment: /dev/dxg is created by WSLg passthrough and is vendor-agnostic, so this also admits Intel/NVIDIA WSL2 hosts into the path the guard exists to keep them out of.

4. It won't merge, and rebasing hard-fails a required check. dash-11 is already taken on main (@id:dash-replay-missing-file-fails-fast, landed in 0595b926 after your merge base). Post-rebase this scenario is #12 while named dash-11, which panics two ordinary cargo test contracts in feature_naming.rs. That's also why your local cargo test --workspace passed — at the merge base it genuinely was #11.

Non-blocking, mostly left inline: the new unit test kills only one of five mutants; two of the four conditions in the poll loop carry no signal (one of them is outright unreachable); the hand-rolled loop gives up the child-exit and reader-panic fast-fail paths that wait_for_screen provides; and a few Gherkin convention points on the scenario. One thing not inline: the PR text's "six unrelated existing unexpected failures" is asserted without evidence — none of the named scenarios have expectations.toml entries, so by the harness's own model they resolve to expected-pass. AGENTS.md:24 asks for verified-vs-assumed to be explicit. Also informational: open PR #399 touches dash_steps.rs too, so there's some future conflict risk there.

Things I checked that are not problems — so you don't spend time on them:

  • Native-Linux detection is unaffected: /dev/dxg never exists there, so the predicate reduces to the old KFD check.
  • The sibling /dev/kfd call sites are not dashboard-facing (main.rs feeds rocm driver status; serve routes through usable_amd_gpu_indices → rocdxg_ready()). Your fix is at the right layer.
  • License manifest needs no action: about.toml sets ignore-dev-dependencies = true, and tempfile 3.27.0 is already in THIRD_PARTY_NOTICES.txt via a non-dev dep.
  • tempfile = "3" matches convention — 6/6 crates use the bare inline form, and there's no [workspace.dependencies] entry.
  • Omitting @requires-os:linux is correct and deliberate; examine.feature and diagnose.feature both document that it can't express "really WSL".
  • The rename is complete, and every consumer of the changed warning strings matches on the "amd-smi unavailable" substring, so nothing breaks.
  • The docs/ci-hardware-testing.md staleness raised in the earlier CHANGES_REQUESTED is a rebase artifact, not an action item — main already made that wording plural in a9937493 (#340). It fixes itself on rebase.

.open(KFD_DEVICE)
.is_ok()
fn gpu_device_accessible(kfd_device: &Path, dxg_device: &Path) -> bool {
device_accessible(kfd_device) || device_accessible(dxg_device)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — this widens a verdict the repo deliberately consolidated, and the widening has a safety edge.

crates/rocm-core/src/lib.rs:2719-2721 already defines the answer: rocdxg_ready() = dxg_device && dxcore && librocdxg && ldconfig_librocdxg. Its doc comment at :2712-2717 says verbatim: "Extracted so serve can act on the same answer examine prints… That split is what let examine report wsl_rocdxg_ready while serve refused on the same machine." docs/ci-hardware-testing.md states outright that /dev/dxg plus dxcore alone are not enough. kfd || dxg is a third, looser WSL verdict, and your own CI box proves the divergence: dxg_device: true, librocdxg: false → this pre-flight passes while examine prints wsl_rocdxg_missing and serve refuses.

The part I'd weigh more heavily: crates/rocm-core/src/lib.rs:2585-2601 documents that /dev/dxg alone proves a host is WSL, because the node comes from WSLg's GPU passthrough — which is vendor-agnostic. So on an Intel or NVIDIA WSL2 box, or an AMD box without ROCDXG, this predicate now passes and hands a real amd-smi the chance to run. I checked the containment: line 124 of this file is Command::new(&self.binary).args(args).output().await with no kill_on_drop anywhere in the file, and crates/rocm-dash-daemon/src/runner.rs:225-232 spawns detection unconditionally at dash startup. tokio::time::timeout abandons the future — it neither kills nor reaps the child — and a D-state process ignores SIGKILL regardless. That's precisely the failure the guard's own doc comment (just above, lines 34-39) exists to prevent, now opened to the population it was widened to admit.

Suggested fix that avoids a new crate dependency: apps/rocm/src/dash.rs:66-71 already calls rocm_core::resolve_amd_smi_binary() and threads the result through RunnerOptions.amd_smi_binary. That same seam can carry a precomputed reachability verdict from rocm-core — one verdict, no new dep, and examine/serve/dash stop disagreeing.

Being honest about the limits of my check: I could not prove that amd-smi actually D-states (rather than merely erroring) on a dxg-present/ROCDXG-absent host, and I couldn't probe /dev/dxg permission bits — there's no such node on my review host. But the pre-flight's own doc comment treats that hang as the reason it exists, so widening it deserves an explicit justification either way.

#[cfg(test)]
mod tests {
use super::*;
#[test]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two small things here.

Nit (formatting): no blank line after use super::*;. This is the sole outlier among 18 test modules repo-wide, and cargo fmt won't catch it.

Coverage: of five plausible mutants of gpu_device_accessible, this test kills only ||→&&. Surviving: returning unconditional true; a typo in DXG_DEVICE; a call site that passes KFD_DEVICE twice; .write(true) instead of .read(true). There's no assertion of the false direction at all, and DXG_DEVICE has exactly one reference in the crate — the untested call site at line 76. Because the test writes a regular file, the interesting real-device failure modes (EACCES / ENXIO / EBUSY on a char device) are unrepresented, so what it mostly proves is std::fs semantics.

Minimal close: add a gpu_device_accessible(missing, missing) == false case, a KFD-only case, and assert_eq!(DXG_DEVICE, "/dev/dxg").

let screen = tui.screen_text();
let nonzero_gpu_count = screen
.lines()
.any(|line| line.contains("GPUs · ") && !line.contains("GPUs · 0"));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — as written this step can only ever time out.

"GPUs · " has exactly one producer: gpu_wall at crates/rocm-dash-tui/src/ui/dock.rs:74, with a single call site at crates/rocm-dash-tui/src/ui/mod.rs:85, guarded by dock::is_wide (dock.rs:30-41) which requires cols >= 180 && rows >= 45. use_detail_size() (tui_driver.rs:305-324) sets 120x40 and is the only resize helper the driver exposes; spawn size is 80x24. So nonzero_gpu_count is unreachable, the loop always exhausts its budget, and on a fully ROCDXG-ready WSL box this panics with "WSL GPU telemetry did not appear" — the fix's only user-facing evidence would report the fix as broken.

Important: the obvious cheap fix is not safe. Dropping to contains("Node throughput ·") && !contains("Unknown GPU") introduces a false-pass path. node_throughput_title (crates/rocm-dash-tui/src/ui/tabs/home.rs:111-134) derives its text from the cached gpu_system_info.gpu_model, which the daemon refreshes only while gpu.is_some() — independently of the per-tick gpus vector (crates/rocm-dash-daemon/src/runner.rs:298-355). A stale cached model name renders Node throughput · <model> with no "Unknown GPU" while live telemetry is failing, which is exactly the state this scenario is supposed to catch.

Two fixes that actually work: add a wide-layout resize helper that reaches dock::WIDE_COLS x WIDE_ROWS and keep the real GPUs · N signal, or assert on a Home-visible zero-GPU indicator instead.

if nonzero_gpu_count
&& screen.contains("Node throughput ·")
&& !screen.contains("Unknown GPU")
&& !screen.contains("amd-smi unavailable")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking, but these two conditions carry no signal.

!screen.contains("amd-smi unavailable") is unreachable, not merely weak: that string is rendered only by crates/rocm-dash-tui/src/ui/tabs/hardware.rs:231-241 on the Observe tab, ActiveTab defaults to Home (crates/rocm-dash-tui/src/app/mod.rs:178-184), and this scenario never switches tabs. It stays dead in the proposed fix for the issue above, too.

And on line 274: "Node throughput ·" is emitted unconditionally by draw_hero_band at any width, so only the paired !contains("Unknown GPU") is doing work there.

{
break;
}
assert!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this hand-rolled poll loop gives up two fast-fail paths that every sibling step inherits from wait_for_screen (tui_driver.rs:416-467) — child-exit detection and PTY-reader-thread panic propagation.

The consequence is specific to this PR: if the daemon crashes in the very code path being changed here, the step spins the full timeout and then reports a misleading "telemetry did not appear" instead of surfacing the crash. Reusing wait_for_screen with a predicate would keep both.

When the user quits the launcher
Then the launcher exits successfully

@id:dash-rocm-29846-wsl-gpu-telemetry @requires-wsl @requires-gpu

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — this scenario executes on no lane, so the coverage the PR description claims doesn't exist.

tests/e2e-cucumber/src/capability.rs:483-486 gates on gfx_target.is_some() && (!is_wsl || driver_status == WSL_DRIVER_READY). For this exact head, the e2e-wsl job (run 34406697117, job 102651494805) reports driver_status: wsl_rocdxg_missing, wsl_dxg_device: true, wsl_dxcore: true, wsl_librocdxg: false, wsl_ldconfig_librocdxg: false, has_amd_gpu: false. The run's own platform.json artifact records this scenario as "expected": "skip", "reason": "requires an AMD GPU; none detected on this host". Skips are filtered out before cucumber counts them (tests/e2e.rs:1219-1254), so the lane reports PASS having never run it — the description's "it will run on the e2e-wsl lane" isn't true today.

AGENTS.md:86,99 accepts a gated lane only if it actually exercises the scenario, and AGENTS.md:98 is explicit that a unit test doesn't discharge the requirement.

In fairness, this isn't something you introduced: the runner's missing librocdxg is a pre-existing problem tracked as issue #223. It explains the gap rather than excusing it, and it's worth saying so in the PR description rather than claiming coverage that won't happen.

Then the launcher exits successfully

@id:dash-rocm-29846-wsl-gpu-telemetry @requires-wsl @requires-gpu
Scenario: dash-11 - Dashboard displays GPU telemetry through the WSL amd-smi backend

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — this won't merge, and the rebase that fixes the conflict hard-fails a required check.

git merge-tree shows conflicts in apps/rocm/src/dash.rs, crates/rocm-dash-daemon/src/runner.rs, and this file. Main already has dash-11 = @id:dash-replay-missing-file-fails-fast, landed in 0595b926 after this PR's merge base.

tests/e2e-cucumber/tests/feature_naming.rs:133-146 and :148-165 are ordinary cargo test contracts: scenario N must be named dash-{N:02} - , and indexes must be unique suite-wide. After rebase this scenario is #12 while still named dash-11, so both tests panic — in a required check. That's also the explanation for your local cargo test --workspace passing: at the merge base it genuinely was #11.

Renumber to dash-12 when you rebase.

Separately, minor wording: the scenario title names the internal backend (amd-smi). No other dash or bench scenario title names a backend — the title is the behavior, not the implementation.

@id:dash-rocm-29846-wsl-gpu-telemetry @requires-wsl @requires-gpu
Scenario: dash-11 - Dashboard displays GPU telemetry through the WSL amd-smi backend
When the user opens the dashboard
Then the dashboard reports live WSL GPU telemetry

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking Gherkin convention points, grouped:

  • This Then step names the transport (WSL). No other dash or bench Then step names a platform — the platform belongs in the tag, the step in behavior terms.
  • On line 127, @id:dash-rocm-29846-wsl-gpu-telemetry is the only ticket-prefixed id among ~115 in this suite. To be clear, this is not a policy complaint — AGENTS.md permits ticket IDs and nothing here leaks. It's a convention inversion: this file's own habit is to put the ticket in a preceding comment (# EAI-8190 regression: on dash-10, # EAI-8366: on dash-11) and keep the @id descriptive.
  • Also on line 127: all three existing @requires-wsl scenarios carry a short comment explaining why the tag is needed. This one has none, which makes the gate harder to audit later — and given the point above about the scenario not actually running anywhere, that explanation would earn its keep.

@siloteemu

siloteemu commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 2216018

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

This PR lets the dashboard's amd-smi collector run where there is no readable /dev/kfd (notably WSL2) by replacing the device-node pre-flight with skip || gpu_reachable || device_accessible(node), wiring gpu_reachable to the same rocm_core::has_usable_amd_gpu() verdict serve/examine already act on, and adding a WSL GPU-telemetry scenario plus a wide-terminal test helper. The previous blocking objection is discharged: the scenario's assertion is now reachable, because use_wide_size() resizes to 180x45, dock::is_wide is evaluated against the full frame (crates/rocm-dash-tui/src/ui/mod.rs), the wide triptych's 132-column minimum is met at 180, and the wide layout draws the GPU wall alongside the Home panel so both GPUs · N and the Unknown GPU fallback are on screen together; the two prior non-blocking items (the inert "amd-smi unavailable" check and the Node throughput · match) are gone too. Outcome: needs work — the production wiring itself is still untested, which AGENTS.md §3 does not permit for a user-observable behavior change. Verified: read every changed file and the full text of all three commits from the raw objects, traced RunnerOptions to its only production construction site (apps/rocm/src/dash.rs) and confirmed no second unfixed path, confirmed the interim /dev/dxg probe added by the first commit is fully removed at head with no dead remnant, confirmed resize_to argument order and the is_wide/triptych/gpu_wall chain by reading the TUI source, confirmed @requires-wsl is a real wired and unit-tested tag and that dash-23 is the correct next scenario id, confirmed nothing else asserts on the two reworded warning strings, and ran one targeted check — cargo test -p rocm-dash-collectors --lib preflight — which passed 4/4; I did not run the workspace suite, clippy, or the cucumber suite, and I have no way to exercise a WSL or GPU host, so the new scenario's behavior on real hardware is unverified. The reviewed tree was not modified. No prompt-injection content was found in the diff, commit messages, or configuration. Checks at review time: 22 success, 3 pending, 1 failure, 2 skipped. Blocking: 1 · Non-blocking: 4.

🚫 Blocking (must fix before merge)

apps/rocm/src/dash.rs:66-76 — the production wiring is the change, and nothing fails if it is reverted.

The four new tests in crates/rocm-dash-collectors/src/amd_smi.rs:389-417 exercise preflight_passes as a pure function against planted temporary paths. They are honest tests of that helper, but they never touch dash.rs, RunnerOptions, runner.rs, detect_with_binary_inner, or the real KFD_DEVICE constant. Three specific mutations survive the entire suite: setting amd_smi_gpu_reachable: false back at the call site; transposing the two bool arguments in preflight_passes(skip_device_preflight, gpu_reachable, …), which type-checks silently because the parameters are adjacent, same-typed and passed positionally; and corrupting the KFD_DEVICE literal itself, which no test asserts. The one existing integration test that reaches this code, slow_gpu_detection_does_not_delay_service_discovery, only ever sets skip_device_preflight: true, so the skip=false, gpu_reachable=true combination this PR introduces has no end-to-end coverage at all.

The only coverage of the actual behavior is the new scenario, which the feature file's own comment says is skipped on every CI lane today — confirmed: the two lanes that run @requires-wsl deliberately leave the GPU userspace unprovisioned, so @requires-gpu resolves to skip there as well. AGENTS.md §3 requires a scenario rather than only a unit test for user-observable behavior, and requires the PR text to name the lane that will exercise it when it is gated — but the gated scenario covers the WSL path only, while the change also alters behavior on bare-metal Linux and Windows (see the first non-blocking item), and those paths have no coverage of any kind.

Concrete fix: add a daemon integration test alongside slow_gpu_detection_does_not_delay_service_discovery in crates/rocm-dash-daemon/tests/end_to_end.rs, constructing RunnerOptions { amd_smi_binary: Some(<fake binary>), amd_smi_skip_device_preflight: false, amd_smi_gpu_reachable: true, ..Default::default() } and asserting the collector produces telemetry. That test fails today on any host without a readable device node, passes with this change, and fails again under the first two mutations above. A sibling case with amd_smi_gpu_reachable: false pins the negative direction and kills the argument-transposition mutant.

Non-blocking

  • crates/rocm-dash-collectors/src/amd_smi.rs:154-157 — the PR text says the change "preserves the existing /dev/kfd guard", but has_usable_amd_gpu() reaches its verdict from KFD sysfs topology and DRM card counts and never opens the device node; sysfs topology is world-readable while /dev/kfd is typically group-gated, so a user outside the render/video group now passes the pre-flight on a host where the device provably cannot be opened, and on Windows the unprobeable-platform true means the OR is unconditional. The behavior is defensible and on the WSL path it is the point of the change, but the prose overstates what the code does — a doc comment stating that the OR deliberately trusts device presence rather than node readability, and why that is acceptable, would settle it.
  • apps/rocm/src/dash.rs:869-905 — the existing runner_options unit tests assert several fields but not amd_smi_gpu_reachable, which is why the field can be reverted silently; even a weak assertion here would pin that it is populated from the probe rather than a literal.
  • tests/e2e-cucumber/tests/e2e/tui_driver.rs:56-59 — WIDE_ROWS/WIDE_COLS duplicate dock::{WIDE_ROWS, WIDE_COLS} by value; a comment naming the source constants would stop the two drifting apart and silently making this scenario assert against a narrow layout again.
  • crates/rocm-dash-daemon/src/runner.rs:404,436 — both reworded warning strings are unasserted by any test, so the wording is free to drift back out of step with the transport-neutral behavior it now describes.

@juhovainio juhovainio changed the title fix(dash): detect amd-smi through WSL DXG fix(dash): detect amd-smi through WSL DXG (EAI-8450) Sep 16, 2026
@juhovainio juhovainio self-assigned this Sep 24, 2026
@juhovainio
juhovainio force-pushed the fix/wsl-dashboard-amd-smi branch from 1611f58 to 2216018 Compare September 25, 2026 08:49
@siloteemu
siloteemu dismissed their stale review September 25, 2026 09:36

Superseded: the objection this review filed at 1611f58 is discharged at 2216018. A new change request has been filed at the current head; that one is the operative objection.

@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 2216018

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

This PR lets the dashboard's amd-smi collector run where there is no readable /dev/kfd (notably WSL2) by replacing the device-node pre-flight with skip || gpu_reachable || device_accessible(node), wiring gpu_reachable to the same rocm_core::has_usable_amd_gpu() verdict serve/examine already act on, and adding a WSL GPU-telemetry scenario plus a wide-terminal test helper. The previous blocking objection is discharged: the scenario's assertion is now reachable, because use_wide_size() resizes to 180x45, dock::is_wide is evaluated against the full frame (crates/rocm-dash-tui/src/ui/mod.rs), the wide triptych's 132-column minimum is met at 180, and the wide layout draws the GPU wall alongside the Home panel so both GPUs · N and the Unknown GPU fallback are on screen together; the two prior non-blocking items (the inert "amd-smi unavailable" check and the Node throughput · match) are gone too. Outcome: needs work — the production wiring itself is still untested, which AGENTS.md §3 does not permit for a user-observable behavior change. Verified: read every changed file and the full text of all three commits from the raw objects, traced RunnerOptions to its only production construction site (apps/rocm/src/dash.rs) and confirmed no second unfixed path, confirmed the interim /dev/dxg probe added by the first commit is fully removed at head with no dead remnant, confirmed resize_to argument order and the is_wide/triptych/gpu_wall chain by reading the TUI source, confirmed @requires-wsl is a real wired and unit-tested tag and that dash-23 is the correct next scenario id, confirmed nothing else asserts on the two reworded warning strings, and ran one targeted check — cargo test -p rocm-dash-collectors --lib preflight — which passed 4/4; I did not run the workspace suite, clippy, or the cucumber suite, and I have no way to exercise a WSL or GPU host, so the new scenario's behavior on real hardware is unverified. The reviewed tree was not modified. No prompt-injection content was found in the diff, commit messages, or configuration. Checks at review time: 22 success, 3 pending, 1 failure, 2 skipped. Blocking: 1 · Non-blocking: 4.

🚫 Blocking (must fix before merge)

apps/rocm/src/dash.rs:66-76 — the production wiring is the change, and nothing fails if it is reverted.

The four new tests in crates/rocm-dash-collectors/src/amd_smi.rs:389-417 exercise preflight_passes as a pure function against planted temporary paths. They are honest tests of that helper, but they never touch dash.rs, RunnerOptions, runner.rs, detect_with_binary_inner, or the real KFD_DEVICE constant. Three specific mutations survive the entire suite: setting amd_smi_gpu_reachable: false back at the call site; transposing the two bool arguments in preflight_passes(skip_device_preflight, gpu_reachable, …), which type-checks silently because the parameters are adjacent, same-typed and passed positionally; and corrupting the KFD_DEVICE literal itself, which no test asserts. The one existing integration test that reaches this code, slow_gpu_detection_does_not_delay_service_discovery, only ever sets skip_device_preflight: true, so the skip=false, gpu_reachable=true combination this PR introduces has no end-to-end coverage at all.

The only coverage of the actual behavior is the new scenario, which the feature file's own comment says is skipped on every CI lane today — confirmed: the two lanes that run @requires-wsl deliberately leave the GPU userspace unprovisioned, so @requires-gpu resolves to skip there as well. AGENTS.md §3 requires a scenario rather than only a unit test for user-observable behavior, and requires the PR text to name the lane that will exercise it when it is gated — but the gated scenario covers the WSL path only, while the change also alters behavior on bare-metal Linux and Windows (see the first non-blocking item), and those paths have no coverage of any kind.

Concrete fix: add a daemon integration test alongside slow_gpu_detection_does_not_delay_service_discovery in crates/rocm-dash-daemon/tests/end_to_end.rs, constructing RunnerOptions { amd_smi_binary: Some(<fake binary>), amd_smi_skip_device_preflight: false, amd_smi_gpu_reachable: true, ..Default::default() } and asserting the collector produces telemetry. That test fails today on any host without a readable device node, passes with this change, and fails again under the first two mutations above. A sibling case with amd_smi_gpu_reachable: false pins the negative direction and kills the argument-transposition mutant.

Non-blocking

  • crates/rocm-dash-collectors/src/amd_smi.rs:154-157 — the PR text says the change "preserves the existing /dev/kfd guard", but has_usable_amd_gpu() reaches its verdict from KFD sysfs topology and DRM card counts and never opens the device node; sysfs topology is world-readable while /dev/kfd is typically group-gated, so a user outside the render/video group now passes the pre-flight on a host where the device provably cannot be opened, and on Windows the unprobeable-platform true means the OR is unconditional. The behavior is defensible and on the WSL path it is the point of the change, but the prose overstates what the code does — a doc comment stating that the OR deliberately trusts device presence rather than node readability, and why that is acceptable, would settle it.
  • apps/rocm/src/dash.rs:869-905 — the existing runner_options unit tests assert several fields but not amd_smi_gpu_reachable, which is why the field can be reverted silently; even a weak assertion here would pin that it is populated from the probe rather than a literal.
  • tests/e2e-cucumber/tests/e2e/tui_driver.rs:56-59 — WIDE_ROWS/WIDE_COLS duplicate dock::{WIDE_ROWS, WIDE_COLS} by value; a comment naming the source constants would stop the two drifting apart and silently making this scenario assert against a narrow layout again.
  • crates/rocm-dash-daemon/src/runner.rs:404,436 — both reworded warning strings are unasserted by any test, so the wording is free to drift back out of step with the transport-neutral behavior it now describes.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 2216018

Blocking: 1. The previous objection — the e2e step's unreachable "GPUs · N" condition at 120x40 — is discharged: use_wide_size() resizes to 180x45, dock::is_wide is evaluated against the full frame, the wide triptych's 132-column minimum is met, and the wide layout draws the GPU wall alongside the Home panel so both strings are on screen together. The two non-blocking items on that step are gone too. Thank you for the fix.

What replaces it is one new finding, in the same class the rest of this PR is careful about.

apps/rocm/src/dash.rs:66-76 — the production wiring is the change, and nothing fails if it is reverted.

The four new tests in crates/rocm-dash-collectors/src/amd_smi.rs:389-417 exercise preflight_passes as a pure function against planted temporary paths. They are honest tests of that helper, but they never touch dash.rs, RunnerOptions, runner.rs, detect_with_binary_inner, or the real KFD_DEVICE constant. Three specific mutations survive the entire suite:

  • setting amd_smi_gpu_reachable: false back at the call site;
  • transposing the two bool arguments in preflight_passes(skip_device_preflight, gpu_reachable, …), which type-checks silently because the parameters are adjacent, same-typed and passed positionally;
  • corrupting the KFD_DEVICE literal itself, which no test asserts.

The one existing integration test that reaches this code, slow_gpu_detection_does_not_delay_service_discovery, only ever sets skip_device_preflight: true, so the skip=false, gpu_reachable=true combination this PR introduces has no end-to-end coverage at all.

The only coverage of the actual behaviour is the new scenario, which the feature file's own comment says is skipped on every CI lane today — the two lanes that run @requires-wsl deliberately leave the GPU userspace unprovisioned, so @requires-gpu resolves to skip there as well. AGENTS.md §3 asks for a scenario rather than only a unit test for user-observable behaviour, and asks the PR text to name the lane that will exercise it when it is gated. The gated scenario covers the WSL path only, while the change also alters behaviour on bare-metal Linux and Windows, and those paths have no coverage of any kind.

Concrete fix: add a daemon integration test alongside slow_gpu_detection_does_not_delay_service_discovery in crates/rocm-dash-daemon/tests/end_to_end.rs, constructing RunnerOptions { amd_smi_binary: Some(<fake binary>), amd_smi_skip_device_preflight: false, amd_smi_gpu_reachable: true, ..Default::default() } and asserting the collector produces telemetry. That test fails today on any host without a readable device node, passes with this change, and fails again under the first two mutations above. A sibling case with amd_smi_gpu_reachable: false pins the negative direction and kills the argument-transposition mutant.

The full round, including four non-blocking items, is in the review comment posted alongside this one.

@juhovainio

Copy link
Copy Markdown
Collaborator

Update

Pushed 6cc865b7, addressing the remaining feedback from both reviewers.

siloteemu (2026-09-25) — the one blocking item was that the apps/rocm/src/dash.rs → RunnerOptions.amd_smi_gpu_reachable production wiring had no end-to-end coverage; the existing tests only ever exercised the amd_smi_skip_device_preflight: true bypass. Added two integration tests in crates/rocm-dash-daemon/tests/end_to_end.rs:

  • gpu_reachable_without_kfd_lets_amd_smi_run_through_the_real_preflight — positive case, drives skip_device_preflight: false, gpu_reachable: true through the real preflight and asserts telemetry is produced.
  • gpu_unreachable_without_kfd_never_runs_amd_smi — negative sibling with gpu_reachable: false, pinning the other direction and killing the argument-transposition mutant called out in the review.

rominf (2026-09-15) — re-verified all four blocking items against current HEAD:

  • e2e layout unreachability (120x40 vs. the wide layout's 180x45 requirement) — already discharged, confirmed by siloteemu's follow-up review.
  • Design/safety concern that the /dev/kfd || /dev/dxg preflight widening invents a new WSL verdict and could admit non-AMD WSL2 hosts (since /dev/dxg is vendor-agnostic WSLg passthrough) — resolved: the preflight no longer checks /dev/dxg itself at all. It now delegates to rocm_core::has_usable_amd_gpu(), which is built on the repo's existing consolidated rocdxg_ready() verdict, so no new verdict is introduced and no vendor-neutral device path is checked directly.
  • dash-11 numeric scenario-ID collision with main — moot: the scenario now uses a descriptive string ID (@id:dash-reports-wsl-gpu-telemetry), not the numeric scheme that collided.
  • Scenario runs on no CI lane — still true, no WSL runner carries a real GPU yet. Tracked separately in [WSL] rocm install driver points at an unshipped repo script instead of installing ROCDXG #223 and called out explicitly in a comment above the scenario, per AGENTS.md §3.

PR description updated to match. Re-requesting review from both.

@juhovainio
juhovainio dismissed stale reviews from rominf and siloteemu September 25, 2026 11:34

All four blocking items resolved as of 6cc865b — see update comment for details.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 6cc865b

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Makes the amd-smi safety pre-flight accept a precomputed GPU-reachability verdict so the dashboard reports telemetry under WSL, renames the pre-flight throughout, adds unit and daemon-integration coverage, distinguishes two amd-smi failure modes in the warning text, wraps the warning panel, and adds a WSL e2e scenario — Needs work: the reachability short-circuit silently widens the hang guard on bare-metal Linux, and the new warning text misdiagnoses the most common failure. Reviewed at check-run conclusions 21 success, 1 failure, 4 pending, 2 skipped. Verified: no build, test or lint was run here — the full suite was not run, and no claim below rests on an execution result; instead I read the pre-flight, its callers, the reachability verdict it now trusts, every added test and the e2e wiring, and confirmed by source that the new scenario identifier is sequential and collision-free, that the harness knows both of its tags, that the wide-layout constants copied into the test driver match the dashboard's own, that the step's assertion cannot be satisfied without real telemetry, that only one production call site constructs the runner options, that the warning panel is rendered in exactly one place, and that a diff-wide scan for internal names, hosts or links came back clean. Blocking: 2 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

crates/rocm-dash-collectors/src/amd_smi.rs:153-160 — the reachability short-circuit disables the kernel-device guard on bare metal, not just on WSL.
The new predicate is a flat three-way OR: skip, or reachable, or the device opens. Because the reachability term is evaluated ahead of the device open, whenever the verdict is true the real device is never opened on any platform. The PR text says the change "preserves the existing kernel-device guard and the protection against uninterruptible amd-smi hangs"; that is accurate for WSL but not for Linux, and nothing in the diff — comment, test or commit message — acknowledges the non-WSL half. Three concrete bare-metal states flip the guard off:

  • The kernel topology directory under sysfs is normally world-readable while the device node is mode 0660 owned by the render group. A user outside that group gets a positive count from sysfs and a permission error from the open — previously refused, now runs the real binary.
  • The DRM fallback in combine_amd_gpu_counts returns a positive count from a display/render-only amdgpu card when no kernel compute node exists at all.
  • has_usable_amd_gpu is documented in crates/rocm-core/src/lib.rs:5165-5175 as deliberately fail-open: an unprobeable platform reports true "so it does not block launches". That polarity is correct for gating a launch and exactly backwards for a hang guard, where unknown must mean "do not run the risky binary". This diff reuses the verdict without inverting that default.
    The guard exists because the process can enter an uninterruptible state that no async timeout can cancel (stated at crates/rocm-dash-collectors/src/amd_smi.rs:36-38), so widening it is a reliability regression, not a cosmetic one. Concrete fix: make the substitution conditional on actually being on WSL — either pass rocm_core::is_wsl_host() && rocm_core::has_usable_amd_gpu() at the wiring site in apps/rocm/src/dash.rs:40-78, or reorder the predicate so the device open is tried first and the verdict is consulted only on a WSL host. Either keeps bare-metal behaviour byte-identical while still fixing WSL, and both are testable with the tempfile pattern the new unit tests already use.

crates/rocm-dash-daemon/src/runner.rs:433-450 — the new "reachable" warning drops the missing-binary explanation and asserts an unsupported GPU instead.
detect_with_binary returns nothing whenever amd-smi version fails for any reason, including the binary not being installed or not resolving. The previous single message named both causes ("no kernel device or binary missing"). The new reachable-branch message names neither, and positively asserts that "this GPU model may not be supported by the installed amd-smi/ROCm, or not supported on WSL yet". On a bare-metal Linux host with a supported AMD GPU and no ROCm runtime installed — a common state, and the one this CLI exists to remedy — the reachability verdict is true and amd-smi is absent, so the user is told their hardware may be unsupported rather than being pointed at the missing runtime. That is a worse diagnosis than the message it replaces. Concrete fix: keep the missing/unresolvable binary as the first-named cause in the reachable branch, e.g. lead with "amd-smi is missing or failed to run" and offer the unsupported-model reading as the secondary possibility.

Non-blocking

  • crates/rocm-dash-daemon/tests/end_to_end.rs — the negative test opens with a hard assertion that the kernel GPU device is unreadable, so it panics on any developer machine that has a GPU; the repo's established idiom for hardware-coupled tests is an #[ignore = "..."] with a reason string (see the manual-hardware test in the collectors crate), which would fail closed instead of red.
  • crates/rocm-dash-tui/src/ui/tabs/hardware.rs:248 — the wrap added to the warning paragraph has no test; reverting it would be caught by nothing, and the file's own render-to-string harness makes an assertion on a wrapped long warning a two-line addition.
  • The head commit message claims the new integration pair kills an argument-transposition mutant; the pre-flight is a symmetric OR of two booleans, so transposing them is behaviour-preserving and no test can kill it — the transposition that would matter is at the options literal in apps/rocm/src/dash.rs, which no test exercises. An assert! on the skip flag in that file's existing options test module would cover the real risk.
  • apps/rocm/src/dash.rs:40-78 — the comment says this is the "same verdict serve/examine already act on", but serve calls usable_amd_gpu_indices directly and I found no examine call site for the helper; the probe is shared by construction, not the call, so the comment overstates the coupling and will mislead the next reader who greps for it.
  • apps/rocm/src/dash.rs:40-78 — on WSL the verdict spawns a subprocess and walks installs synchronously inside an async function; it is once per dashboard launch and matches the adjacent binary-resolution precedent, so it is not a per-tick cost, but it is worth a word in the comment.

@juhovainio

Copy link
Copy Markdown
Collaborator

Pushed 602624a addressing @siloteemu's latest review:

  • Blocking: gated amd_smi_gpu_reachable on rocm_core::is_wsl_host() at the dash.rs wiring site, instead of passing the fail-open has_usable_amd_gpu() verdict unconditionally. That verdict is deliberately fail-open on any unprobeable platform, which was the wrong polarity for this pre-flight's hang guard — it silently skipped the mandatory /dev/kfd open on any host, not just WSL. This restores byte-identical bare-metal behavior.
  • Blocking: fixed the reachable-but-failing warning message to keep the missing/unresolvable-binary explanation instead of asserting the GPU model is unsupported, which misdiagnosed the common bare-metal case of ROCm/amd-smi simply not being installed.
  • Non-blocking: the negative preflight integration test now skips itself (rather than asserting red) on a dev machine with a real, accessible /dev/kfd, matching the repo's existing #[ignore] idiom for hardware-coupled tests.
  • Non-blocking: added a test pinning the WSL-gated call site in dash.rs, and a test covering the GPU warning paragraph's line-wrap behavior in the TUI.
  • Non-blocking: corrected doc comments that overstated what the existing daemon integration tests actually exercise (the preflight_passes OR is symmetric, so swapping its two bool args is behavior-preserving and untestable at that layer).
  • Non-blocking: noted the WSL verdict subprocess spawn is per-launch, not per-tick.

Full workspace test suite is clean except the pre-existing /opt/rocm-coupled rocm-core failure on this dev box (unrelated to this change, present on main).

@juhovainio
juhovainio dismissed siloteemu’s stale review September 25, 2026 13:23

Addressed in 602624a — see PR comment for details.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 602624a

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Lets the dashboard's amd-smi safety preflight accept a WSL GPU transport in place of a readable /dev/kfd, renames the KFD-specific seams, and reworks the dashboard's GPU warnings; outcome: Needs work. Verified: on a scratch copy I mutation-tested each production line the new tests name — the TUI wrap test correctly dies when .wrap(Wrap { trim: false }) is removed, and each of the three OR terms in preflight_passes, dropped separately, is killed by exactly one unit test (a clean truth table); but runner_options_gates_gpu_reachable_on_wsl still passes when the is_wsl_host() && gate — the entire fix for the prior round's first blocking finding — is deleted. I also confirmed the production gate itself is correct (is_wsl_host() is not fail-open, so bare-metal behaviour really is unchanged), that only one production site constructs RunnerOptions, and that the leak scan is clean. Not verified: the full suite, the e2e suite, anything on WSL or GPU hardware, and the new Gherkin scenario (it is tagged so it skips on every CI lane). Checks at publication time: 25 success, 2 failure, 2 skipped; both failures are on lanes that are red on the trunk itself at this commit, so neither is attributable to this diff. Blocking: 2 · Non-blocking: 4.

Prior round: the first blocking finding is fixed in production code (the WSL gate is real and correctly placed) but its test does not hold it; the second is fixed in the branch it was raised against and regressed in the sibling branch (see below). Prior non-blocking items (i), (ii) and (v) are genuinely addressed — the negative test now skips at runtime instead of asserting red, the wrap has a test that works, and the once-per-launch cost is documented (and is now short-circuited away entirely on bare metal).

🚫 Blocking (must fix before merge)

1. apps/rocm/src/dash.rs:904-917 — the only test guarding the WSL gate survives the exact regression its doc comment names.

The assertion re-computes the production expression verbatim:

assert_eq!(
    opts.amd_smi_gpu_reachable,
    rocm_core::is_wsl_host() && rocm_core::has_usable_amd_gpu()
);

opts.amd_smi_gpu_reachable is literally assigned that same expression at line 81, so the test can only fail on a host where the mutated and unmutated expressions disagree. I deleted rocm_core::is_wsl_host() && from line 81 on a scratch copy and ran cargo test -p rocm --bin rocm runner_options_gates_gpu_reachable_on_wsl: ok, 1 passed. That holds on every GPU-less Linux host — the probe returns an authoritative empty set, so both operands are false and the gate's removal is invisible. The comment's own parenthetical ("on any host where the fail-open verdict fires") concedes this without drawing the conclusion: the fail-open verdict fires on non-Linux targets, not on the Linux lanes where this crate's unit tests actually run against real hardware conditions. So the fix for the prior round's hang-guard finding ships with no effective test on its primary platform.

The comment also claims the test "catches an argument transposition at this call site". It cannot: A && B over two side-effect-free bools is commutative, so there is no behavioural mutant to catch. This is the same incorrect claim the prior round raised about a commit message — and the head commit corrected it in crates/rocm-dash-daemon/tests/end_to_end.rs:538-540 ("swapping its two bool arguments is behaviour-preserving and not something an integration test at this layer can catch either way") while simultaneously reintroducing it here. Two comments in one diff say opposite things about the same mutant.

Fix — use the pattern this diff already established one crate over. Extract the gating into a pure helper and truth-table it, exactly as preflight_passes(skip, reachable, path) was extracted:

/// The verdict may only substitute for the device open on WSL: `has_usable_amd_gpu()`
/// is fail-open on an unprobeable platform, which is the wrong polarity for a hang guard.
fn gpu_reachable_for_preflight(is_wsl_host: bool, has_usable_gpu: bool) -> bool {
    is_wsl_host && has_usable_gpu
}

then assert (false, true) -> false (the bare-metal fail-open case the finding was about), (true, true) -> true, (true, false) -> false. That kills the un-gating mutant on every platform, with no dependence on what the test host's hardware happens to be. Keep a thin call-site test if you want, but drop the mirrored assert_eq! and delete the transposition sentence.

2. crates/rocm-dash-daemon/src/runner.rs:452 — the fix to the reachable branch regressed the sibling branch, dropping the missing-binary cause the base branch named.

Base text: "amd-smi unavailable (no /dev/kfd or binary missing)". New text: "amd-smi unavailable (GPU device inaccessible or probe failed)".

The reachable branch (line 446) now correctly leads with "is missing, unresolvable, or failed to run" — that part of the prior finding is properly fixed. But because amd_smi_gpu_reachable is now gated on is_wsl_host(), every bare-metal host falls into the else branch, including the precise state the prior finding was filed about: a supported GPU, a readable /dev/kfd, and no ROCm/amd-smi installed. In that state the device was accessible — that is how control reached this line at all — so the first-named cause is provably false, and the one true cause (the binary is not installed) is no longer named anywhere. The base branch named it. The remediation moved the explanation from the branch that gets it on bare metal into the branch that never fires there.

The wording is also now pinned by crates/rocm-dash-daemon/tests/end_to_end.rs:... ("must not fall back to the bare-metal inaccessible message"), so CI will certify it.

Fix — restore the missing/unresolvable binary as a named cause in the else branch, e.g. "amd-smi unavailable (not installed, unresolvable, or the GPU device is inaccessible)", and update the test's exact-string comparison to match.

Non-blocking

  • crates/rocm-dash-collectors/src/amd_smi.rs:55-56 — the doc claim that threading the verdict through means "serve/examine/the dashboard never disagree about whether a host's GPU is usable" is no longer true once the caller gates on WSL: on bare metal serve acts on the ungated verdict (including the DRM fallback) while this preflight consults only the device open, so they can disagree. Scope the sentence to WSL, or it will mislead the next reader the same way.
  • crates/rocm-dash-daemon/src/runner.rs:436-449 — the comment asserts amd_smi_gpu_reachable "is only ever true on WSL", and the warning text hardcodes "(WSL ROCDXG bridge)", but RunnerOptions is public and this crate cannot enforce that; the file's own tests set the flag true off-WSL. Either rename the field to say what it means here, or drop the WSL-specific phrasing from the message.
  • Commit 602624a's message says the negative test was made to match "the repo's existing #[ignore] idiom", but it uses a runtime eprintln! + early return. That is a legitimate existing idiom for a condition only knowable at runtime — the code is right, the message describes something else.
  • The new Gherkin scenario is tagged such that it skips on every CI lane today; the feature file discloses this honestly, but the PR's verification summary should say plainly that the scenario is not exercised anywhere in CI rather than listing it alongside the test runs that did execute.

@juhovainio
juhovainio force-pushed the fix/wsl-dashboard-amd-smi branch from 602624a to 26ee33b Compare September 30, 2026 12:07
@juhovainio

Copy link
Copy Markdown
Collaborator

Pushed 26ee33b addressing @siloteemu's latest review (602624a):

  • Blocking Let Lemonade auto-select its llama.cpp backend #1 — truth-tabled the WSL gate: extracted a pure gpu_reachable_for_preflight(is_wsl_host, has_usable_gpu) helper in apps/rocm/src/dash.rs and pinned it with an explicit truth-table test, so a GPU-less Linux host stays gated even if the is_wsl_host() check were deleted.
  • Blocking Enable native-certs for ureq across all crates #2 — restored the missing/unresolvable-amd-smi-binary cause in runner.rs's bare-metal warning branch (dropped by the earlier reachable-branch fix), matching the suggested wording exactly, with the pinned assertion in end_to_end.rs updated to match.
  • Non-blocking: scoped the "never disagree" doc-comment in amd_smi.rs to WSL specifically (the crate doesn't itself depend on rocm-core, so this only holds because apps/rocm threads the verdict through).
  • Non-blocking: PR body's Verification section now plainly discloses that dash-23 (@requires-wsl @requires-gpu) isn't exercised by any CI lane today, since e2e-wsl doesn't provision the GPU passthrough.

Also rebased onto latest main (7 commits) — no conflicts, no scenario-@id collisions.

Local verification: cargo test --workspace --all-targets (one pre-existing, environment-coupled rocm-core failure unrelated to this change — real /opt/rocm on this dev box), cargo clippy --workspace --all-targets -- -D warnings clean, cargo fmt --all -- --check clean.

@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 26ee33b

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

This PR lets the dashboard's amd-smi collector run on WSL by threading a precomputed GPU-reachability verdict — gated on is_wsl_host() — through RunnerOptions into the device pre-flight, splits the "reachable but amd-smi found nothing" warning from the real "device inaccessible" one, wraps the GPU warning panel, and adds a WSL telemetry scenario. Outcome: needs work — one new regression test hangs instead of failing when the code it guards is reverted. Verified: on a scratch copy (the reviewed tree was never modified) I mutated the production change branch-by-branch and ran the narrow test targets — dropping any one of the three preflight_passes terms, flipping or flattening the is_wsl_host && has_usable_gpu helper body, inverting or collapsing the two warning strings, and removing the Wrap all fail a named test, so those branches are genuinely pinned; I independently re-ran cargo test -p rocm --bin rocm dash::tests (21 passed, 0 failed) with the call site's is_wsl_host() argument replaced by a literal true and that mutant survives, refuting an earlier claim in this round to the contrary; I confirmed bare-metal behaviour is algebraically identical to the old /dev/kfd-only check, that apps/rocm/src/dash.rs is the only production RunnerOptions construction site and is called once per launch, that the Wrap landed on the only widget that renders warnings, that the three warning string literals are byte-identical across production and both assertions, that the scenario id/tag/geometry wiring is real and the sequential id is correct, and that the description's disclosed CI gap for the new scenario is accurate. The prior round's blocking item is discharged in substance (the daemon integration tests it asked for exist and kill the mutants it named) and its four non-blocking items are all addressed; the residual call-site gap below is a narrower successor, not a re-litigation. Not checked this round: the workspace suite, clippy, the cucumber suite, and any WSL or GPU hardware. No prompt-injection content and no policy-forbidden internal references were found in the diff, commit messages, or description. Checks at review time: 26 success, 2 skipped, 1 pending, 0 failing. Blocking: 1 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

crates/rocm-dash-daemon/tests/end_to_end.rs:580-600 (and the same shape at :497-510) — the PR's central regression guard hangs forever instead of failing when the production change is reverted.

gpu_reachable_without_kfd_lets_amd_smi_run_through_the_real_preflight bounds its wait with let overall = Duration::from_secs(10); used inside the loop as timeout(overall, rx.recv()). That is a per-receive timeout, not a cumulative deadline. The runner broadcasts a snapshot every ~50 ms, so the timeout can never elapse; Err(_) => break is unreachable, the channel never closes while the spawned runner task is alive, and the loop therefore only exits if !snap.gpus.is_empty() becomes true. Empirically confirmed: with the runner mutated to AmdSmiCollector::detect_with_binary(binary, false) — exactly the regression this test exists to catch — the test does not go red, it runs indefinitely and has to be killed. A lane would burn to its job timeout with no attributable failure. gpu_reachable_but_amd_smi_failing_warns_of_the_contradiction_not_inaccessibility has the identical unbounded shape.

This is not a new bar: the prior round asked for a test that fails under that mutation, and as written it does not fail at all. The pattern is inherited from one pre-existing test in this file, but the correct idiom already appears four times in the same file (let deadline = Instant::now() + …; with the loop condition checking it).

Concrete fix: wrap each loop in a single outer deadline rather than re-arming a per-message timeout, e.g. let found = timeout(overall, async { loop { … } }).await.unwrap_or(false); with the handle.abort() / final assert! left outside — or adopt this file's existing deadline-and-Instant::now() idiom. Either bounds total wall time, so the mutated build fails red at 10s. (Checked against the remedy itself: the outer-timeout form still aborts the runner task afterwards, still yields false on expiry so the existing assertion message fires unchanged, and does not alter the passing path's timing.)

Non-blocking

  • apps/rocm/src/dash.rs:74-77 — the helper's truth table pins the body, but nothing pins the binding: replacing rocm_core::is_wsl_host() with a literal true here passes all 21 dash tests, so the WSL gate's wiring is still unguarded, and the description's "deleting the is_wsl_host() && gate now fails the test on every platform" reads broader than what the code guarantees — a reader will trip on this again; a test that self-skips on a WSL host and otherwise asserts !runner_options(..).amd_smi_gpu_reachable closes it without re-deriving the production expression.
  • crates/rocm-dash-daemon/tests/end_to_end.rs:477-481 — the comment says "force the fake through the real detection path", but amd_smi_skip_device_preflight: true routes to the test-only bypass that hardcodes gpu_reachable: false; amd_smi_gpu_reachable: true alone already satisfies the pre-flight, so dropping the skip flag would make the comment true and mirror production.
  • crates/rocm-dash-daemon/src/runner.rs:81-83 — the field doc still says "Only the daemon integration test that points amd_smi_binary at a deliberately-slow fake script flips this"; two tests now flip it.
  • crates/rocm-dash-daemon/tests/end_to_end.rs:544 — unlike its negative sibling at :616, the positive test has no "skip if /dev/kfd is readable" guard, so on a GPU-equipped host it passes through the device-node arm and proves nothing about the reachability arm it is named for.
  • crates/rocm-dash-daemon/src/runner.rs:354 — the None => AmdSmiCollector::detect() arm silently discards amd_smi_gpu_reachable (detect() hardcodes false); unreachable today since the only production caller always sets the binary, but it is the same gate at a second site and will be wrong the first time someone leaves amd_smi_binary: None.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 26ee33b

Requesting changes on one item. It was verified by mutation on a scratch copy; the checked-out tree under review was never modified.

The change's central regression guard hangs indefinitely instead of failing when the production change is reverted.

The test bounds its wait with a ten-second duration that is applied inside the receive loop, so it is a per-message timeout rather than a cumulative deadline. The runner broadcasts a snapshot roughly every 50 ms, so that timeout can never elapse: the error arm that breaks the loop is unreachable, the channel never closes while the spawned runner task is alive, and the loop can only exit once the condition it is waiting for becomes true. Confirmed empirically — with the runner mutated to the exact regression this test exists to catch, the test does not go red; it runs indefinitely and has to be killed. A CI lane would burn to its job timeout with no attributable failure, and the sibling test asserting the contradiction warning has the identical unbounded shape.

This gates because a test that cannot fail for the defect it names does not guard anything, and the previous round asked specifically for a test that fails under that mutation. It is not a new bar being introduced late. The shape is inherited from one pre-existing test in the same file, but the correct idiom already appears four times in that same file: a single deadline computed once, with the loop condition checking it.

To resolve: wrap each loop in one outer deadline rather than re-arming a per-message timeout — an outer timeout around the whole loop, with the task abort and the final assertion left outside it — or adopt the deadline-and-elapsed idiom the file already uses. Either bounds total wall time so the mutated build fails red at ten seconds.

We ran the same adversarial pass over that proposed remedy: the outer-timeout form still aborts the runner task afterwards, still yields a false result on expiry so the existing assertion message fires unchanged, and does not change the timing of the passing path.

For the record, the previous round's blocking item is discharged in substance — the integration tests it asked for exist and do kill the mutants it named — and all four of its non-blocking items are addressed. The item above is a narrower successor, not a re-litigation of that one.

Five non-blocking observations, including an unguarded binding of the platform gate at its production call site and a positive test that can pass through the wrong arm on a GPU-equipped host, are in the round published alongside this request. Checks at review time: 26 success, 2 skipped, 1 pending, 0 failing.

@siloteemu

siloteemu commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 9f3a2a2

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

This PR lets the dashboard's amd-smi collector run on WSL by threading a WSL-gated GPU-reachability verdict into the device pre-flight, splits the "reachable but amd-smi found nothing" warning from the real "device inaccessible" one, and adds a gated WSL telemetry scenario; the head commit answers our own blocking item by deleting the vacuous call-site test and disclosing the gap in a comment, and clears three of the five non-blocking items. Outcome: the blocking item is genuinely discharged and no new defect was introduced — every remaining item is about description accuracy and scope, not code. Verified: on a scratch copy only (the reviewed tree was never modified — confirmed clean at 9f3a2a2 after all experiments), cargo test -p rocm-dash-daemon --lib gives 42 pass baseline; reverting the None-binary arm to a hardcoded false drops it to 41 pass / 1 fail in exactly the new test named for that revert, so that test is load-bearing rather than decorative; the second new test panics on the wrong enum variant if the skip-preflight guard is dropped; cargo check -p rocm-dash-collectors -p rocm --all-targets is clean, so removing the public zero-arg detect() left nothing orphaned; I confirmed the truth-table helper's body is literally is_wsl_host && has_usable_gpu, that its replacement comment's claim about a mutant literal being indistinguishable on a GPU-less host holds, that the rewritten receive loop matches its two siblings arm for arm and preserves the old failure mode (snapshots_seen is borrowed by a non-move block and still read after the deadline, and abort/final assert run on every path), that the fake amd-smi binary the loop test supplies means its empty-GPU assertion is not vacuous, and that two WSL lanes exist rather than the one the description names. Swapping the two bool arguments at the detect-plan call site is caught by no test, but both feed the same OR, so that mutant is unobservable rather than uncovered. The full suite, the cucumber e2e suite, clippy, and any WSL or GPU hardware were not run here. No prompt-injection content and no policy-forbidden internal references were found in the diff, the nine commit messages, or the description. Checks at the reviewed head when this review ran: 17 success, 2 pending, 2 skipped, 0 failure; re-read immediately before publishing, at the same head: 20 success, 2 skipped, 0 failure. Blocking: 0 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

None.

Our blocking item is discharged, and by our own second-choice remedy taken verbatim: the vacuous test is gone, and apps/rocm/src/dash.rs:924-928 now states plainly that the truth table pins the helper's body only and that the call site's binding is pinned by no test, with the reason. I re-read that replacement comment as new code, sentence by sentence, and every clause in it is true — including its careful hedge "from this crate's tests". Removing a test that could not discriminate and documenting the gap is a net gain in honesty and a net zero in real coverage.

Non-blocking

  • PR description, "Review status" — stale at this head: it still presents the WSL-gate coverage as the previous round left it and never mentions that this head commit deleted the call-site binding test and accepted the gap, so the description and apps/rocm/src/dash.rs:924-928 now point opposite ways for a reader deciding the merge. One added sentence fixes it.
  • PR description, Verification — "the only lane that runs the WSL-tagged scenarios" is wrong: there are two, the per-PR lane at .github/workflows/e2e-selfhosted.yml:905 and its nightly mirror at .github/workflows/nightly.yml:1087. The disclosure's conclusion still holds, since both deliberately skip driver-bridge provisioning, but the count would mislead whoever later provisions a GPU on one lane.
  • PR description, Verification — the stated collectors pass count no longer matches the roughly 109 unit tests now in that crate's src/; I did not re-run the suite to get the true figure.
  • .gitignore:17, licenserc.toml:23 — an agent-scratch directory from the author's local tooling (deliberately not repeated here) is added to shared repo config purely to satisfy a local pre-commit hook, unrelated to this PR's subject; the adjacent entry is precedent, though the new one uses a trailing slash where its neighbour does not.
  • crates/rocm-dash-daemon/tests/end_to_end.rs:406 — all three sites this PR owns now use the single-outer-deadline shape, but one pre-existing re-armed per-message timeout remains here, with the same hang-instead-of-fail-red hazard; out of scope, noted because the head commit's stated aim was to retire the class.

Confusion worth preventing. Last round we called the description's sentence about deleting the WSL conjunction false. Re-reading it, that phrase names the helper body's &&, where it is true; the call site has no && at all, only an argument. That was reviewer error caused by a genuinely ambiguous phrase naming an expression that exists in two forms, and a competent reader will trip on it again. The cheap prevention is to say "the helper's conjunction" in that sentence and add the one clause about the call-site binding — which folds into the first item above.

Prior round's items

  • Blocking — vacuous call-site binding test for the WSL gate: discharged, via the second remedy we offered (drop the test, state the gap honestly above the helper).
  • Non-blocking 1 — re-armed per-message timeout in the unreachable-GPU regression guard: discharged; it now matches its two siblings arm for arm and preserves the old failure mode.
  • Non-blocking 2 — the None-binary arm's reverted verdict caught by nothing: discharged, and proven by mutation — the extracted pure function's new unit test fails on exactly that revert.
  • Non-blocking 3 — zero-arg detect() with one ignored caller: discharged beyond the suggestion; it was removed outright, its sole caller updated, and the workspace still compiles with all targets.
  • Non-blocking 4 — the unenforced "on bare metal the caller always passes false" convention: not addressed; those lines are byte-identical to the previous round.
  • Non-blocking 5 — hand-copied terminal-size constants in the e2e TUI driver: not addressed; that file has no diff since the previous round.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 67648a3

Requesting changes on one item. Everything below was verified by mutation on a separate scratch copy; the checked-out tree under review was never modified.

First, what landed well. The previous round's blocking item is fully discharged, and discharged properly. The two receive loops now carry a single outer deadline instead of a re-armed per-message timeout, and that is load-bearing rather than decorative: with the production wiring reverted, the guard now panics at its own ten-second deadline and the filtered run finishes on its own in about fourteen seconds instead of hanging until a job timeout. The task abort and the final assertion still execute, the early-close path still breaks out rather than burning the deadline, and the passing path's timing is unchanged. Four of the five non-blocking items from that round are also genuinely discharged, each checked individually rather than taken on trust.

The one item that does not hold is the one we ourselves proposed, and the fault is ours.

apps/rocm/src/dash.rs:928-946 — the new test added to pin the call-site binding of the platform gate passes vacuously on the hosts it actually runs on, and the comment above it states the opposite.

The test skips only when the host is WSL, then asserts that the reachability flag is false. But the gate is a conjunction of the WSL check and a usable-GPU check. On an ordinary non-WSL runner with no usable GPU, the second term is already false, so both the real expression and the mutant evaluate to false and the assertion holds either way. Confirmed empirically: with the call site's platform check replaced by a literal true, the dash target still reports 22 passed, 0 failed, and a diagnostic run on the same host shows both terms false. The test can only distinguish the two branches on bare metal with a GPU visible to the kernel driver.

This gates under the standing rule that a test which cannot fail for the defect its own name and comment claim to catch does not guard anything. It is aggravated rather than mitigated by the doc comment directly above it, which says the test pins that the real platform check is passed in rather than a literal, and by the description's claim that deleting the gate now fails the test on every platform. Both sentences are false on the platform where the test will actually run, and a silent green that reads as coverage is worse than the acknowledged gap it replaced.

Naming our own error plainly: the previous round proposed this exact remedy — a test that self-skips on a WSL host and otherwise asserts the flag is false — and did not run an adversarial pass over that proposed remedy before recommending it. The remedy was wrong. The author implemented it faithfully and in good faith. This is our error, not theirs, and we are flagging it only because the resulting test now makes an inaccurate coverage claim that a later reader would rely on.

To resolve, either:

  • make the skip honest by adding the usable-GPU check as a second precondition, so the test asserts only where it can discriminate. We ran the adversarial pass over this one: it does kill the mutant on GPU hardware, but it means the test never exercises anything on ordinary CI, so it is documentation-grade coverage only — say so in the comment rather than leaving the present claim standing; or
  • drop the test and replace the sentence above the helper with an honest one: that the helper's body is covered by its truth table, but its call-site binding is not pinned by any test.

We also checked and advise against a third option that looks attractive: extracting an inner helper taking the two booleans as parameters only moves the unpinned binding into the thin outer wrapper.

Five non-blocking observations are in the round published alongside this request, including one remaining instance of the same unbounded-timeout shape the head commit fixed in its two siblings.

Checks at review time: 24 success, 2 pending, 2 skipped, 1 neutral, 0 failing.

@siloteemu
siloteemu dismissed their stale review September 30, 2026 17:13

Superseded by a new change request filed at the current head. The blocking item this review named — the two receive loops bounding their wait with a re-armed per-message timeout — is DISCHARGED, and discharged well: the outer-deadline rewrite was re-verified by mutation on a scratch copy this round, and the reverted build now fails red at its own deadline instead of hanging. Four of the five non-blocking items are discharged too. The new request carries one item, and it is our own error rather than the author's: the remedy this round proposed for the remaining item does not work, because the test we suggested passes vacuously on any host without a usable GPU. That is stated plainly in the new request, with two workable alternatives. Only the newest request is operative; this one is retired so the PR shows one live concern rather than two.

michaelroy-amd and others added 9 commits October 1, 2026 09:10
Signed-off-by: Michael Roy <michael.roy@amd.com>
…flight

The device pre-flight guards against amd-smi hanging in uninterruptible
D-state on a host with no usable GPU. The previous WSL fix probed /dev/dxg
directly, but that device is vendor-agnostic (present under WSLg's GPU
passthrough for Intel/NVIDIA too), so it could pass on hosts where the real
amd-smi binary would still hang, and it disagreed with the WSL readiness
check serve/examine already use (rocm_core::has_usable_amd_gpu, which
consults WslSummary::rocdxg_ready()).

Thread that same precomputed verdict through RunnerOptions instead of
teaching rocm-dash-collectors a second, weaker WSL check. apps/rocm already
depends on rocm-core and already resolves the amd-smi binary path the same
way, so it computes has_usable_amd_gpu() once and passes it down alongside
the binary; rocm-dash-collectors/rocm-dash-daemon stay free of the
dependency.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
The step over-claimed what it checked ("WSL GPU telemetry") when the
assertion itself is platform-agnostic; renamed to match, mirroring the
scenario title already dropping "amd-smi" for the same reason.

Replaced the hand-rolled poll loop with the existing wait_for_screen_where
helper, dropped the "Node throughput ·" check (present even in the
Unknown-GPU fallback, so it never discriminated) and the "amd-smi
unavailable" check (that warning only ever renders on the Hardware tab,
which this scenario never opens, so it was unreachable), and removed the
stale ROCM-29846 panic-message prefix. Kept the real nonzero "GPUs · N"
check and the "Unknown GPU" negative check, and resize to the dock's wide
layout (use_wide_size, new helper mirroring use_detail_size) so the GPU
panel the assertions read is actually the one rendered at that geometry.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
rocm-core's has_usable_amd_gpu() treats WSL ROCDXG bridge readiness as
proof a usable GPU exists, but on unsupported GPU models (e.g. pre-Strix
APUs like gfx1103) the bridge can be ready while amd-smi still fails
outright. The dash previously surfaced a generic "device inaccessible"
warning for this case, directly contradicting what examine had just
reported. Surface a distinct message when amd_smi_gpu_reachable was true
but amd-smi still found nothing, naming the actual contradiction instead.

Also wrap the GPU warning panel's Paragraph so long warning text wraps
instead of getting clipped at the panel width, matching every other
Paragraph in the TUI.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
Add integration tests that exercise amd_smi_gpu_reachable through the
real device pre-flight (amd_smi_skip_device_preflight: false), the
combination the prior WSL fix's production wiring relies on but that
no test previously drove: every existing test used the skip-preflight
bypass, which never calls preflight_passes's gpu_reachable OR-branch.

A positive case proves a fake amd-smi actually produces telemetry when
gpu_reachable is true and /dev/kfd is absent; a negative sibling with
gpu_reachable: false pins the other direction and kills the
argument-transposition mutant between the two bools.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
Address siloteemu's PR review findings:

- Gate the amd-smi device-preflight bypass on is_wsl_host(), not just
  has_usable_amd_gpu(). That helper is deliberately fail-open on any
  unprobeable platform (so it never blocks a launch) — the wrong
  polarity for a hang guard, where an unknown verdict must not skip the
  real /dev/kfd open. Restricting the substitution to WSL keeps
  bare-metal behaviour byte-identical to a plain /dev/kfd check.
- Fix the reachable-but-failing warning message to keep the
  missing/unresolvable-binary explanation instead of jumping straight
  to "unsupported GPU model", which misdiagnosed the common bare-metal
  case of ROCm/amd-smi simply not being installed.
- Make the negative preflight integration test skip itself (instead of
  asserting red) on a dev machine with a real, accessible GPU device,
  matching the repo's existing #[ignore] idiom for hardware-coupled
  tests.
- Add coverage for the WSL gating call-site and for the GPU warning
  paragraph's line-wrap behavior, and correct doc comments that
  overstated what the existing tests exercise.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
The only test guarding `is_wsl_host() && has_usable_amd_gpu()` re-derived
that same expression as its assertion, so it passed unchanged on a
GPU-less Linux host even with the gate deleted. Extract a pure
`gpu_reachable_for_preflight` helper and truth-table it instead.

Since that gate now routes every bare-metal host into runner.rs's
`else` branch, restore the missing/unresolvable-amd-smi-binary cause
the base branch named there, which the reachable-branch fix had
dropped.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
The two integration tests bounding a 10s wait re-armed the timeout inside
the receive loop instead of around it, so a snapshot arriving every ~50ms
kept the timeout branch unreachable: a reverted gpu_reachable wiring made
the loop hang forever instead of failing red. Wrap each loop in a single
outer timeout so total wall time is actually bounded.

Also address the review's non-blocking findings: pin the WSL-gate call
site's binding (not just gpu_reachable_for_preflight's body) with a new
test, correct a stale comment/skip-flag mismatch, add the missing
/dev/kfd skip guard to the reachability-arm test's positive sibling, and
thread amd_smi_gpu_reachable through the None (PATH-lookup) arm of
run_loop's amd-smi detection instead of silently discarding it.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
…dual gaps

The call-site binding test added last round passes vacuously on every host
without a real, kernel-visible AMD GPU (i.e. ordinary CI), since the second
conjunct of the gate is already false there regardless of the first. Drop it
and say so honestly in the truth-table test's comment instead, per the
reviewer's own adversarially-checked recommendation.

Also closes the round's non-blocking items: give the remaining unbounded
per-message receive loop the same single-outer-deadline shape as its two
siblings, fold the now single-callered zero-arg detect() into its one
remaining (hardware-only, ignored) caller, and extract the amd-smi detect
plan selection into a pure function so a revert of the None-binary arm's
threaded gpu_reachable verdict is actually caught by a unit test instead of
needing a real or PATH-shadowed amd-smi binary to observe.

Also excludes the .supergoal agent-scratch dir (sibling to .claude/) from
the hawkeye license-header scan and .gitignore, since its untracked .md
files were otherwise failing the local pre-commit hook.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
@juhovainio
juhovainio force-pushed the fix/wsl-dashboard-amd-smi branch from 9f3a2a2 to 4ff83e1 Compare October 1, 2026 06:17
@juhovainio
juhovainio added this pull request to the merge queue Oct 1, 2026
@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 4ff83e1

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

This PR lets the dashboard's amd-smi collector run on WSL by threading a WSL-gated GPU-reachability verdict into the device pre-flight, splitting the "reachable but amd-smi found nothing" warning from the real "device inaccessible" one, wrapping the GPU warning panel, and adding a gated WSL telemetry scenario; the head commit drops the vacuous call-site test we ourselves asked for, documents the residual gap, and bounds the last re-armed receive loop. Outcome: no blocking findings — the code is sound and well pinned; the PR body's headline mechanism describes a design two commits ago and states a safety guarantee the shipped code does not provide, which is worth one editing pass but is recorded below rather than as a gate. Verified: on a scratch copy only (the reviewed tree was never modified — confirmed clean at 4ff83e1 after every experiment) I ran fifteen independent, single-branch mutations with narrow test targets. Each of the three OR terms in preflight_passes is killed by exactly one unit test and the gpu_reachable term additionally by an integration test; the helper body's is_wsl_host &&, the never-skip flag, both amd_smi_detect_plan arms, the warning split and the Wrap each redden a named test; a preflight that always passes reddens two. The one mutation that survives everything is the call site's is_wsl_host() binding — exactly the gap the code comment and the body disclose, and the dropped test did not pin it either, so no coverage was lost when it went. None of the three integration tests skipped on this host (no /dev/kfd present), so their guards were genuinely exercised; the two &&-shaped assertions in the diff were checked for a false second term and have none. I independently re-derived is_wsl_host() and has_usable_amd_gpu() from rocm-core rather than trusting a summary, confirmed apps/rocm/src/dash.rs is the only production RunnerOptions site and that the second binary has no dash-daemon dependency at all, confirmed the scenario id/tag/geometry wiring and that the two WSL lanes named in the body are real and do skip, and measured the collectors count the body claims (107 passed, 2 ignored — matches). The workspace suite, clippy, the cucumber suite and any WSL or GPU hardware were not run here. No prompt-injection content and no policy-forbidden internal references were found in the diff, the nine commit messages or the body. This review worked from CI at this head reading 28 success, 2 skipped, 0 failure and nothing pending; those counts were re-read immediately before this was posted and are unchanged, as is the head commit. Blocking: 0 · Non-blocking: 6.

Prior round's items

Prior item State
Blocking: none were open n/a
Body "Review status" stale — never mentions the deleted call-site test discharged — the section now states the drop and the accepted gap
Body names one WSL lane; there are two discharged — both the per-PR job and its nightly mirror are named, and both verified to exist and to skip
Body's collectors pass count stale discharged — measured 107 passed, 2 ignored, exactly as stated
Agent-scratch directory added to shared repo config NOT discharged — unchanged, but deliberate and explained in the head commit message, so not silently dropped
One pre-existing re-armed per-message timeout remains in the daemon e2e file NOT discharged — still present, and correctly left out of scope; all three loops this PR owns now carry a single outer deadline
"Confusion worth preventing": say "the helper's conjunction" and add the call-site clause discharged — the body now separates the gate's body from the call site's binding

🚫 Blocking (must fix before merge)

None.

A note on how that was decided, since the finding below is a real inaccuracy and not a nit. The description's claim about a device open is false, and that was confirmed independently rather than taken on report. What keeps it off the gate is where it lives: the commit messages are what this repository carries onto its default branch when a pull request is squashed, and they are accurate — the second commit's message states in terms that the earlier approach probed the device node and that the reachability verdict replaced it. So the permanent record describes the shipped design correctly, and the stale text is confined to the pull request page. Measured against what merging actually leaves behind, that is an editing pass rather than a merge gate. It is still worth making before this lands, because this page is where someone revisiting the hang-guard question will look, and the sentence as written invites them to believe a guard exists that does not.

Non-blocking

  • PR description, Summary bullet 1 and the "Why" paragraph — they describe the design of the first commit, not the one being merged. Bullet 1 says the change accepts "readable /dev/dxg" as the WSL transport for the amd-smi safety preflight, and "Why" says production invokes amd-smi "only when /dev/kfd or /dev/dxg can be opened". The shipped code does neither: the second commit replaced the direct device probe with the rocm-core verdict, crates/rocm-dash-collectors/src/amd_smi.rs opens /dev/kfd and nothing else, and on WSL the pre-flight is satisfied by the reachability verdict alone, so no device is opened at all before amd-smi is invoked. The underlying predicate is not an open either: rocm_core::is_wsl_host() tests for the existence of /dev/dxg or a kernel build string, and the WSL path additionally requires a ready ROCDXG bridge — so the real gate is stricter than the sentence in one direction and looser in another. The body also contradicts itself, since its fourth paragraph describes the actual gated-verdict design correctly. Suggested fix: drop the word "readable" and the "can be opened" clause, and say instead that on WSL the reachability verdict substitutes for the /dev/kfd open, that /dev/kfd remains the only device this crate opens, and that the verdict requires both a WSL host and a ready ROCDXG bridge. Deleting the stale bullet and sentence outright, leaving the fourth paragraph to stand alone, is cheaper and equally accurate.
  • crates/rocm-dash-daemon/src/runner.rs:457-480 — flipping the warning branch so every host gets the WSL-ROCDXG-flavoured message is caught by no test; only the contradiction branch has a positive assertion, and the bare-metal string appears solely in a negative != check that passes trivially if the production text changes.
  • crates/rocm-dash-daemon/tests/end_to_end.rs:540 — "amd_smi_skip_device_preflight: false here, unlike the tests above" is now half true: of the two tests above, only the slow-detection one sets it true. Small comment rot introduced by the previous round's fix.
  • crates/rocm-dash-collectors/src/amd_smi.rs:61-64 — "On bare metal the caller always passes false" remains a convention this crate cannot enforce on a public field; carried unchanged from an earlier round and still only a documentation guarantee.
  • crates/rocm-dash-collectors/src/amd_smi.rs:23 — the KFD_DEVICE literal is asserted by nothing, so corrupting it stays green across every target; pre-existing rather than introduced here, and unobservable on a host with no /dev/kfd anyway.
  • .gitignore:17, licenserc.toml:23 — an agent-scratch directory from local tooling is still added to shared repo config, unrelated to this PR's subject; the head commit explains why, and the adjacent entry is precedent, so this is a scope note rather than an objection.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 1, 2026
@juhovainio
juhovainio added this pull request to the merge queue Oct 2, 2026
@juhovainio
juhovainio removed this pull request from the merge queue due to a manual request Oct 2, 2026
@juhovainio
juhovainio added this pull request to the merge queue Oct 2, 2026
Merged via the queue into main with commit d37cfba Oct 2, 2026
31 checks passed
@juhovainio
juhovainio deleted the fix/wsl-dashboard-amd-smi branch October 2, 2026 12:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants