Repository navigation
ci(e2e): GPU preflight asserts no leaked engine process survives - #672
Conversation
The GPU preflight's free-memory floor is a fixed 8 GiB (Strix) or 16 GiB (discrete) applied to whatever pool rocm-smi reports, so what it detects is set by the pool's SIZE and not by what a serve holds: against a ~62 GiB Strix GTT aperture it trips only past ~54 GiB held, and against a 192 GiB MI300X only past ~176 GiB. A leftover llama-server holding 10-20 GiB — the exact thing the step exists to catch — clears every lane. Add `reclaim-gpu.sh --assert-clear`: reusing select_leaked, it exits non-zero naming any leaked E2E engine process still running, and kills nothing. The preflight runs it after the memory floor is satisfied, so a card still draining from the reclaim step's kills is absorbed by the bounded wait rather than reported. Keep the floor. The two gates are complementary, not alternatives: the process check catches the leftover serve the floor misses, and the floor catches a non-E2E tenant exhausting a shared host, which no process check can see. Extract the six near-identical native Linux preflight bodies into one .github/scripts/gpu-preflight.sh, so this landed once rather than six times (advances #294). The Strix/APU variant was a strict superset of the discrete one — the two-pool query degrades to the single-pool one, and a discrete card's VRAM total never reaches the carveout branch — so the only per-lane input is GPU_PREFLIGHT_MIN_FREE_GIB. No lane's effective floor changes. The two PowerShell blocks keep memory-only semantics and say so; a Win32_Process equivalent is follow-up. Closes: #446 Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
The extraction in the previous commit rests on the APU preflight body being a strict superset of the discrete one: a discrete card's VRAM total is far above any floor so the carveout branch is unreachable, and the two-pool `--showmeminfo vram gtt` query degrades to the single-pool call the discrete lanes always made. The existing cases cover a rocm-smi that ACCEPTS the two-pool form on a discrete card, and one that rejects it on a carveout. Nothing covered a rejection on a full-size pool — the arm that actually has to fall back on an Instinct host with an older rocm-smi. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…prose
The shape assertion searched the whole script for `--assert-clear`, which the
header comment and the error message both name. Replacing the
`if bash "${RECLAIM_SCRIPT}" --assert-clear; then` condition with `if true; then`
left the test green — it was matching the explanation of the gate rather than
the gate. Filter comment lines out first.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
The two WSL bodies keep only the free-memory floor: they do not run `reclaim-gpu.sh --assert-clear`, because they execute inside the WSL distro through Invoke-WslBash.ps1, where the repo checkout is not at a path the step can assume. Their comment still claimed a leftover serve "still fails", which is the overclaim #446 exists to correct and which the native lanes now actually deliver. State the weaker guarantee and why, rather than leaving the native and WSL wording identical when the gates no longer are. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Follow-up in That wording was accurate before this PR, when every lane had the same (coarse) gate. It is not accurate now: the six native Linux lanes gained Leaving the native and WSL comments identical while the gates diverged would have re-created the defect this PR fixes, one layer down. The comments now state the weaker guarantee and why, and point at the Windows gap as the companion follow-up. Verified on Linux at |
jussielo-amd
left a comment
There was a problem hiding this comment.
Review — needs work (minor)
Full review of the whole change. Implements the process-check gate from ROCMAI-446 / #446 in full; separately advances #294 (preflight-body deduplication) without closing it — the PR body scopes this accurately as "advances," not "closes."
Reviewed via 4 independent file-group passes (core scripts; workflow YAML; contract tests + docs; agent-instruction/history/comments) plus a consolidation pass and a targeted design-review check on self-hosted runner concurrency. Diff range dfe1df3aff0974cc58ef2be5cf5b6a276fbd98d3...492c4e73362e8d45d6e511e9cad232c38c67a367 (PR head at review time). All 25 CI checks were green at review time.
Blocking (minor, no functional consequence)
- New doc prose cites a tracked file as a bare backtick path instead of a relative markdown link —
docs/testing.md:1192
AGENTS.md §5 asks for file citations added/edited in markdown to be relative links rather than bare backtick paths, so a future rename/move doesn't go stale silently. This PR's new paragraph cites the new.github/scripts/gpu-preflight.shas`.github/scripts/gpu-preflight.sh`rather than as a link. Mitigating: the entire file already cites every other file this same bare way (12+ pre-existing instances — zero markdown-style links anywhere indocs/testing.md), so this continues an established pattern in this doc rather than introducing a new one, and CI's lychee job has nothing to check either way. Fix: wrap the new citation in a relative link, e.g.[`gpu-preflight.sh`](../.github/scripts/gpu-preflight.sh); or, if bare-citation style is intentional for this doc, worth noting as an explicit exception.
Non-blocking / for your consideration
- Escalating the leaked-process check to a hard CI-failing assertion inherits
select_leaked's existing unanchored-substring over-match exposure at a higher cost. Previously a false match meant an extra kill; now it fails the whole job. The code's own comment already acknowledges this tradeoff directly, so flagging only so it's a conscious sign-off rather than implicit. - The real discrete-lane 16 GiB floor is asserted present in the workflow YAML only by a static string check; no dynamic
GateCaseexercises a near-16-GiB-boundary scenario. All current discrete fixtures have huge margins, so the floor's specific value never changes a verdict today. A boundary case (~10–15 GiB free) would close the gap. - No contract test pins that all six native-Linux call sites invoke the shared script identically, the way the PowerShell/WSL twins are pinned. Low risk today (each site is a one-line delegation), but there's no structural guarantee against future re-inlining drift.
- Self-hosted runner concurrency was checked directly: each GPU lane targets a distinct hardware
runs-onlabel, and a GitHub self-hosted runner executes one job at a time, so the new process gate can't race against a concurrent job on the same host within this workflow. No finding here — noted since it was the main open design question worth checking.
Positive signals
assert_clear()deliberately reusesselect_leaked()rather than re-implementing matching — avoiding exactly the kind of matcher drift that caused the original six inlinepkill -fcopies to diverge from their PowerShell mirrors.- The new contract test
the_shared_preflight_asserts_no_leaked_engine_process_survivesstrips comment lines before matching the gate, with a comment documenting it was tightened after an earlier version matched the script's own prose about the gate rather than the gate itself — independently reproduced via the same mutation during this review. field()'s|| truecorrectly flips from cosmetic insurance (inline step, nopipefail) to load-bearing (standalone script,set -euo pipefail) — a subtle shell-semantics trap, called out explicitly in a comment rather than silently inherited.
Deployment notes
A leaked-process false positive (see non-blocking item above) now fails a self-hosted CI job outright instead of being silently reclaimed — worth a heads-up to whoever triages self-hosted lane failures, since the failure signature changes.
This is a feedback-only review comment and does not block the merge.
The existing delegation test asks whether each lane MENTIONS the shared script. It cannot see a lane that keeps the delegation and grows a line of its own above it — one more `pkill -f`, a lane-local retry — which is how the original six inline copies drifted from their PowerShell mirrors in the first place. Compare the six bodies to each other, as the PowerShell/WSL twins already are, then pin the shape they agree on: a lone `bash <script>` line. Equality alone passes six identically re-inlined copies; the shape check alone misses five of six. Also drive the discrete lanes' 16 GiB floor across its own boundary. Every other discrete fixture clears or misses it by over a hundred GiB, so the number never decided a verdict; at 15 GiB free it must fail and at exactly 16 it must pass. And cite the new script as a relative link rather than a bare backtick path (AGENTS.md §5), so lychee catches a rename. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Thanks — this was a useful review. Addressed in 1. Bare backtick path (blocking, minor) — fixed
I took the link rather than declaring an exception. AGENTS.md §5 ends with "This covers citations you add or edit, not converting existing bare paths in an unrelated change" — which both obliges the new citation and scopes the fix, so the other 12 pre-existing bare paths are deliberately left alone. This one points at a file the PR creates, which is where a silent rot costs the most. 2. Nothing pinned that the six call sites delegate identically — fixed, and the gap was realNew Mutation evidence (all on the devbox,
The first is the decisive one: the pre-existing test stays green against exactly the re-inlining drift you flagged. The new failure names it — 3. No near-16-GiB
|
juhovainio
left a comment
There was a problem hiding this comment.
Reviewed this (closes #446, partially advances #294) with a correctness pass plus a separate Standards/Spec pass. CI is fully green, and the core approach - a second gate asserting no leaked E2E engine process survives, layered after the existing free-memory floor - is sound and well-reasoned; I traced the two-gate ordering, the set -euo pipefail semantics change, the xtask contract-test rewrites, and the new reclaim-gpu.sh --self-test case, and all of it holds up.
Requesting changes on one concrete gap, left as an inline comment below: every rocm-smi call in the new gpu-preflight.sh is wrapped in timeout 15 specifically because the driver can hang, but the new --assert-clear call that implements this PR's own headline gate has no such bound. If the process scan ever stalls, this step has no internal ceiling and rides to the job's own timeout - exactly the failure mode the rest of the script exists to avoid.
Also worth noting, not blocking:
- One structural fact rather than a defect: the self-hosted GPU lanes that actually exercise
--assert-cleararen't part of per-PR CI at all, so this PR's own run validates the new behavior only indirectly (xtask contract tests + the bash--self-testharness), never on live hardware - matches the PR description's own disclosure. - Against #294 specifically: the duplicated
reclaim-gpu.shinvocation across the 8 jobs is untouched by this PR (only the shared preflight block was deduplicated), and #294's acceptance criterion for a lane-count-vs-docs contract test isn't implemented here (docs/testing.mdis touched,docs/ci-hardware-testing.mdis not). Also, this PR uses a shared shell script rather than #294's proposed composite GitHub Action - a reasonable deviation, just flagging it as one.
None of the non-blocking items need to hold this up; the timeout gap does, since it's a real hang risk in the gate this PR adds.
| # and self-tests them against decoy processes. A second matcher in CI is how | ||
| # the original `pkill -f` patterns drifted. | ||
| if bash "${RECLAIM_SCRIPT}" --assert-clear; then | ||
| exit 0 |
There was a problem hiding this comment.
Every other external call in this script (rocm-smi, 4 call sites above) is wrapped in timeout 15 because the driver can hang. This call - the new assert-clear gate this PR adds - isn't. If reclaim-gpu.sh --assert-clear's process scan ever stalls (e.g. a wedged ps/driver query inside it), this step has no bound and rides to the job's own timeout instead of failing fast the way the rest of this script is designed to. Worth wrapping in timeout <n> the same as the rocm-smi calls above, with a clear error message on the timeout-exit-code (124) path.
There was a problem hiding this comment.
Fixed in 1cde744 (+ test in 63cacde) — you were right that this was more than a style inconsistency, so the comment now says why.
What is bounded. timeout 15 bash "${RECLAIM_SCRIPT}" --assert-clear, same style and same value as the four rocm-smi calls above. The two --report-holders diagnostics got the same bound: having just established that the scan can stall, an unbounded diagnostic on the failure path would spend the ceiling the bound just saved.
What a timeout now does, and why that direction. It fails the gate, with its own message, as a third outcome distinct from clear/survivor:
::error::GPU preflight failed: the leaked-process scan (reclaim-gpu.sh --assert-clear) did not
finish within 15s and was timed out — the host could NOT be shown clear. …
A scan that could not complete is not evidence the host is clean, so passing on it would convert a hang into a silent green on exactly the hosts #446 is about. The status is captured (|| assert_rc=$?) rather than tested with if, so under set -euo pipefail the 124 path is reached and named rather than collapsing into the survivor message.
The reason the bound matters more here than for rocm-smi is now in the script: select_leaked walks /proc/[0-9]*/cmdline and reads each entry, and a process wedged in uninterruptible sleep can block that read — which is the state a wedged GPU process is in. The hang risk is correlated with the condition being detected, not independent of it.
Evidence the bound actually fires (devbox, real gpu-preflight.sh, --assert-clear stubbed to exec sleep 60, fixture rocm-smi showing a free aperture):
GPU ready: 61 GiB GTT free (>= 8 GiB).
STUB reclaim-gpu.sh called: --assert-clear
::error::GPU preflight failed: the leaked-process scan … did not finish within 15s …
STUB reclaim-gpu.sh called: --report-holders
--- exit=1 elapsed=15s ---
Control, same stub against b65b697: --- exit=0 elapsed=60s ---. Unbounded it did not merely hang — it rode the stall out and then passed the gate. Clear host still exits 0 in 0s; a named survivor still exits 1 with the unchanged #446 message.
Pinned, not just measured once. preflight_bounds_the_leaked_process_scan drives the real script against a never-returning scan and asserts exit 1, the timeout wording, that holders are still reported, and that it ends at the bound (the stub sleeps 60s, so elapsed separates "cut short" from "happened to return"). The shape test gained the complementary pins (timeout 15 … present; 124 handled distinctly). Both mutations are caught: dropping the timeout fails both tests after 60s; changing -eq 124 to -eq 125 fails both at 15s with the "did not say so" message.
I did not add a reclaim-gpu.sh --self-test case: the bound lives in gpu-preflight.sh, not in the reclaim, and the self-test's job is the matching rules. The xtask harness is where the preflight's --assert-clear behaviours are already pinned, and it was free — cargo test -p xtask is 20.09s without the new case and 20.10s with it, since it runs in parallel with the existing multi-second cases.
devbox: 256 xtask tests pass (255 + the new one), reclaim-gpu.sh --self-test green including all four --assert-clear cases, cargo clippy --workspace --all-targets -D warnings clean, bash -n + shellcheck clean. The self-hosted GPU lanes still cannot be exercised from here, as the PR body discloses.
Your two non-blocking items are deliberately left as they are: (1) the live-hardware validation gap — this round adds one more off-hardware proof but does not close it; and (2) the #294 deltas (duplicated reclaim-gpu.sh invocation untouched, no lane-count-vs-docs contract test, docs/ci-hardware-testing.md untouched, shared script rather than a composite action). The PR body still says "advances" rather than "closes" for #294.
Every rocm-smi call in gpu-preflight.sh runs under `timeout 15` because the driver can hang. The `--assert-clear` gate this PR adds had no such bound, so the one step added to stop a job hanging to its cap could itself hang to it. The risk is not hypothetical and not independent of what is being detected: `select_leaked` reads /proc/<pid>/cmdline for every process, and a holder wedged in uninterruptible sleep can block that read — which is the state a wedged GPU process is in, i.e. exactly the condition the gate fires on. The bound FAILS the gate. `timeout` reports expiry as 124, which is handled as its own outcome with its own message: a scan that could not finish is not evidence the host is clear, and reporting it as either a pass or a named survivor misdescribes it. The `--report-holders` diagnostics are bounded the same way — having just established the scan can stall, an unbounded diagnostic would spend the ceiling the bound just saved. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
A timeout nobody has exercised is a timeout nobody knows works, and this one can never be exercised from per-PR CI on live hardware: the lanes that run `--assert-clear` for real are self-hosted and nightly. So the reclaim stub gains a mode that never returns, and the native preflight is run against it: the run must fail, say the scan did not finish within 15s, still name what it could see holding the card, and END AT THE BOUND. The stub sleeps 60s, so the elapsed time is what separates "the bound cut it short" from "the scan happened to return" — unbounded, that case takes a full minute and then PASSES the gate, because the stub exits 0. `exec sleep`, not a plain one: the bound's SIGTERM has to land on the sleep itself, or it is orphaned still holding the captured stdout pipe and the harness waits out the stub rather than timing the script. The shape test gains the complementary pin — the call is wrapped in `timeout 15` and 124 is handled distinctly — so removing either is named immediately instead of by a case that takes a minute to go red. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
tomastola
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 63cacde
This is an approval filed at the explicit request of the engineer driving this review, overriding the automation's usual rule of never filing one. The review itself found nothing blocking.
Review — no blocking findings
Full review of the whole change, read at 63cacde0: the original 6-file change reviewed at 492c4e73, plus the three commits that landed after it (b65b6972, 1cde744, 63cacde0), which were reviewed in full.
The description names Closes #446. Advances #294. The change does what those two issues ask: #446 option 3, a process-based assertion with the floor kept, and #294 for the six native Linux preflight copies. The description predates the three later commits and does not mention them.
Blocking
None
Non-blocking
None
Decisions for the author
- The description does not cover the three commits added after it — non-blocking-improvement
1cde744adds a behaviour change:gpu-preflight.shnow bounds--assert-clearand--report-holderswithtimeout 15, and fails with its own message when the scan times out (exit 124). The description's "What this adds" and "Everything load-bearing is preserved" sections, and its verification numbers (cargo test -p xtask— 254 passed), predate it. The docs were updated and the behaviour is pinned by tests; only the description is stale. Add a line for it. --assert-clearturns the reclaim's substring over-match into a red lane instead of a kill — tradeoff
The assertion sharesselect_leakedwith the reclaim, which is the benefit: no second matcher to drift. The cost is that any process whose command line contains anENGINE_MARKERSorE2E_ROOTSsubstring now fails the lane. Only$$/PPIDare excluded. Confirm a failure is wanted over a warning; I think it is, because that is the point of the gate. Confidence 40.- Concurrent matrix legs on one host now fail visibly instead of being killed silently — tradeoff
nightly.ymle2e-gpu-nightly-strixhasmatrix: channel: [release, nightly]. If two legs can share a host, one leg's live serve already got SIGTERMed by the other's reclaim, and can now also fail the other's preflight. The exposure is not new; only the failure mode changed. I could not establish whether they share a host. - The 15s bound relies on
timeoutkilling the scan — non-blocking-improvement
timeout 15sends SIGTERM only, with no-k. A process truly wedged in uninterruptible sleep ignores SIGTERM until its syscall returns, so the bound is firm for killable waits and best-effort for a hard D-state hang. Adding-k 5would make the escalation explicit. The comment claims the ceiling for the whole class. Not verified on real hardware. Confidence 30. - The native-lane floor sits in step
env:, outside what the drift test compares — tradeoff
Swapping the floor between two lanes of the same class is not caught. The newevery_native_linux_preflight_delegates_in_the_same_shapetest compares the bodies to each other and pins the lone-delegation shape, and it states plainly that the floor is deliberately not part of it. The per-lane value is still pinned byevery_native_linux_preflight_delegates_to_the_shared_script. Confidence 30. - Gate 2 checks once, with no retry — non-blocking-improvement
After the floor is met,--assert-clearruns a single time. The reclaim step SIGKILLs and does not wait for exit, so a process still dying could fail the lane. A separate step sits between them andprocess_alivetreats zombies as dead, so exposure is small. Confidence 20. - Multi-GPU hosts: gate 1 reads only the last GPU's row — non-blocking-improvement
fieldends intail -1. Pre-existing and documented in a comment; gate 2 is process-based and has no such blind spot.
Positive signals
- The new timeout is tested by running it.
preflight_bounds_the_leaked_process_scandrives a stub that sleeps 60s against the 15s bound and asserts the exit code, the specific message and the elapsed time. Usingexec sleepso the signal lands on the sleep itself rather than an orphan holding the pipe is the detail that makes the test time the bound and not the stub. - The 16 GiB floor is now driven across its own boundary. The 15 GiB and exactly-16 GiB cases pin both the value and
>=against>. Previously every discrete fixture cleared or missed the floor by over a hundred GiB, so the number decided nothing. - A timeout fails the gate with its own reason. A scan that could not finish is not treated as a clear host, and the exit 124 is captured rather than tested with
if, so clear, survivor and never-finished stay distinguishable. - The anti-drift test compares the six bodies to each other and also pins the shape. Either half alone would be satisfied by six identical re-inlined copies or would miss one of six.
Deployment notes
- No self-hosted GPU lane has run against this head. The first real run of the process gate and of the new 15s bound will be the first self-hosted run after merge, against real
rocm-smioutput and a real/proc. Watch that run. - The assertion can newly fail lanes on hosts with a leaked or over-matched engine process. That is the intended behaviour.
- The floors are unchanged: 16 GiB on the four discrete lanes, 8 GiB on the two Strix lanes.
- The Windows blocks still check memory only. The PR documents this and pins it with a test.
- At review time, CI on
63cacde0was still running. Required checks should be green before merge.
What this covered
I read all 6 changed files completely at 63cacde0, against the merge base. The change was read in two parts: the original diff at 492c4e73, then the incremental 492c4e73..63cacde0 (three commits touching .github/scripts/gpu-preflight.sh, docs/testing.md and xtask/src/workflow_contract.rs), the latter by me directly rather than a worker.
What I ran, from a clean git archive of the head:
cargo test -p xtask workflow_contract: 60 passed.bash scripts/reclaim-gpu.sh --self-testand the original script experiments, on the earlier head.- Mutations on the new head, all caught: dropping the
timeouton--assert-clear(2 tests red), swallowing the 124 status (2 red), changing the 124 branch (2 red), and-geto-gtin the floor comparison (1 red). - On the earlier head, also:
if true,|| true, and the floor changed in each workflow, all caught.
Did not run: shellcheck and shfmt (not installed here); the self-hosted GPU lanes; the Windows and WSL blocks, which are unchanged apart from comments, so I read them only; the history, prior-changes and instruction-adherence passes as separate workers.
Not reconciled against any review threads or earlier discussion. The Jira ticket named in the original request (ROCMAI-446) concerns the coverage gate and does not correspond to this change, which names GitHub issues #446 and #294.
Addressed at 63cacde. You were right, and the measurement was worse than the report: with --assert-clear stubbed to stall, the unbounded version at b65b697 exited 0 after 60s — it did not merely ride to the job timeout, it PASSED the gate, asserting a clean host it had never verified. Now timeout 15 bash "${RECLAIM_SCRIPT}" --assert-clear, with 124 handled as a distinct third outcome that fails with its own ::error:: ("could NOT be shown clear"), separate from both pass and survivor-found. Same stub now exits 1 at 15s. Both mutations die: dropping the timeout fails at 60s, 124 -> 125 fails at 15s. Your two non-blocking items are deliberately left (no live-hardware validation; the #294 deltas).
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 63cacde
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Round note: every finding below is new (first round on this PR). This round files a change request on the second and third blocking findings (contract tests that stay green with the guarded code mutated or deleted, and new code paths no test reaches). The first blocking finding (self-hosted GPU lanes not yet run: E2E self-hosted run 38563490556 is pending with zero jobs) is missing evidence, not a defect in the change; it does not gate on its own. The description finding alone would not gate either, because the squash commit is built from commit messages.
Review — needs work
Full review of the whole change.
Closes #446 (adds the process-based option 3 from that issue and keeps the floor, as the issue suggests) and does the preflight part of #294; the rest of #294 (report upload, reclaim, bootstrap dedup) is not attempted, and the PR says "Advances" rather than closes, which is accurate.
Blocking
-
(new) The required self-hosted GPU lanes have not run for this head —
E2E self-hostedrun 38563490556
For63cacde0this workflow ispendingwith zero jobs. Only the hostedCIandCodeQLruns completed. Theservepath filter matches this PR (scripts/**,.github/workflows/**), so the GPU lanes are expected to run but have not. The PR's main deliverable is a script that gates every native Linux GPU lane, and so far it has only run against stubbedrocm-smiand stubbed reclaim. Four of its six callers arenightly.ymllanes that never run on a PR.
Confidence 100 · architectural · Fix: get the self-hosted run to actually start (cancel the stalled run and re-run), and dispatchnightly.ymlon this branch before merge. -
(new) Contract tests claim guarantees they cannot fail on —
xtask/src/workflow_contract.rs:1265,:1281-1300,:1361-1375, andis_apu_preflightat:1181
Each of these was mutation-tested, and all tests stayed green:- The floor check is
step.contains("GPU_PREFLIGHT_MIN_FREE_GIB: 16"). A substring match passes for160. - The floor check and
is_apu_preflightboth match comment text inside the step. Replacing an APU lane'senv:key with# GPU_PREFLIGHT_MIN_FREE_GIB: 8passes, and that lane then silently runs at the script's 16 GiB default. That is exactly what the assertion message says it prevents ("silently inherit the script's default"). every_native_linux_preflight_delegates_in_the_same_shapecompares only therun:body. Its rationale names "a lane-local retry, a second floor" as the drift it guards against, yet each of these passes on a single lane:continue-on-error: true,if: false,GPU_PREFLIGHT_CEILING_SECS: 1.continue-on-error: truewould make a required gate advisory.- In
the_shared_preflight_asserts_no_leaked_engine_process_survives, the first assertioncode.contains("--assert-clear")can never fail. Comment lines are filtered out, but the non-commentecho "::error::… (reclaim-gpu.sh --assert-clear) did not finish…"atgpu-preflight.sh:246still contains the string. The test's own comment (":1361") warns of exactly this, and theif bash … --assert-clear; thenshape it describes has not existed since 1cde744. With the call deleted, the test fails on a different assertion (thetimeoutliteral), not on the one claiming "no longer RUNS".
Confidence 100 (each mutation was run) · logic · Fix: parse the step's
env:mapping and compare the floor value exactly; pin the non-run:keys of the six steps as equal; make the assert-call check match the call itself (timeout 15 bash "${RECLAIM_SCRIPT}" --assert-clear). - The floor check is
-
(new) New code paths no test reaches —
scripts/reclaim-gpu.sh(--assert-clear)arm inmain),.github/scripts/gpu-preflight.sh:247-248, :254, :267- The xtask harness always stubs reclaim. The self-test (check 14) calls
assert_cleardirectly. So nothing exercisesbash reclaim-gpu.sh --assert-clearas the preflight invokes it. With the dispatch arm removed, every test stays green. The script then exits 2, which the preflight reports as "a leaked E2E engine process is still holding it (listed above)" with nothing listed. Exit 127 (script not found) gets the same message. - Removing the newly added
timeout 15from either--report-holderscall also leaves every test green. The stubs return instantly, so no test can observe these bounds, even though the script's comments argue they are load-bearing.
Confidence 100 · logic · Fix: have one harness case run the real
reclaim-gpu.sh --assert-clear(scoped throughRECLAIM_SELFTEST_SCOPEto a decoy); give the "leaked process" message only for exit 1 and a generic message for any other non-zero exit; add a slow--report-holdersstub case. - The xtask harness always stubs reclaim. The self-test (check 14) calls
-
(new) The PR description misstates the change — PR body
- The "Falsified" section describes a mutation of
if bash "${RECLAIM_SCRIPT}" --assert-clear; then. That code no longer exists. - Two later commits (1cde744, 63cacde) are not mentioned. They introduce a new failure mode: a
--assert-clearscan that does not finish within 15 s now fails the lane with its own message. - The body also ends with a "Generated with Claude Code" footer. AGENTS.md §11 says "avoid AI-generated boilerplate footers".
Confidence 85 · mechanical · Fix: update the description to cover the bounded-scan behaviour and the current mutation evidence, and drop the footer.
- The "Falsified" section describes a mutation of
Non-blocking
-
(new) Gate 2 runs once, and the argument for running it only after the floor does not hold on the discrete lanes —
.github/scripts/gpu-preflight.sh:205-260,scripts/reclaim-gpu.sh(reclaim, the SIGKILL loop)- The script comment reads: "By here the memory is back, so anything the reclaim rules still match is a process that survived being reclaimed." The PR's own table says the floor only trips near exhaustion (past ~176 GiB held on MI300X). So on those lanes, passing the floor says nothing about whether a process reclaim just SIGKILLed has finished exiting.
reclaim()returns right after sending SIGKILL, without waiting for the process to exit.- On a single sighting, gate 2 exits 1 with no retry inside the remaining ceiling.
- Separately, the scan forks
tronce for every/procentry. On this dev box (~900 pids, load ~28)--assert-cleartook 16–19 s. A full preflight with a healthy stubbed GPU and no leak then failed with the 124 message, which names a D-state holder as the cause even though no process was stuck. - CI exposure is not measured; native hosts that see the full host
/proc(the nightly Strix lane) are the likeliest to hit it.
Confidence 70 · logic · Fix: poll gate 2 until the existing deadline, as gate 1 does; read cmdlines with a builtin (
read -r -d ''from/proc/$pid/cmdline) instead of forkingtr; word the 124 message as one possible cause, not the cause. -
(new) WSL comments edited by this change contradict their own step —
.github/workflows/e2e-selfhosted.yml:1091-1095, :1231-1233;.github/workflows/nightly.yml:~1171-1176, :1309-1311- The new comment says the script is not invoked because "the repo checkout is not at a path this step can assume". The same body runs
(cd /root/work/rocm-cli && bash scripts/reclaim-gpu.sh --report-holders). The code is what governs, so--assert-clearcould run there the same way. The comment also sets up the stated follow-up on a false premise. - The unchanged lines saying "the other preflight blocks are plain
run:steps …bash -e {0}… no pipefail" are now stale, because those lanes runset -euo pipefailthrough the script. - A dangling line,
# \rocm-smi` itself isn't, is left ate2e-selfhosted.yml:1095`.
Confidence 85 · mechanical · Fix: correct the reason given (or run
--assert-clearfrom the clone), and update the pipefail sentence. - The new comment says the script is not invoked because "the repo checkout is not at a path this step can assume". The same body runs
Decisions for the author
-
Discrete lanes now run the APU superset — tradeoff
The four discrete bodies were the older single-pool version. They now:- try the two-pool query first;
- require the
(B):unit; - print a pools line;
- catch a zero total.
On a wedged driver, each poll costs about 35 s instead of 20 s, so the 90 s ceiling gets about 3 attempts instead of 5 (measured with stubs). One body for six lanes is the point of #294. The cost is fewer retries on a wedged discrete card, so confirm that is acceptable.
-
Shared script rather than the composite action #294 proposes — tradeoff
The script keeps job headers untouched and can be tested from a scratch tree. A composite action would also absorb the reclaim, upload and bootstrap duplication #294 measures. The script is a reasonable first step; just confirm that #294 stays open for the rest.
Positive signals
--assert-clearreusesselect_leakedrather than adding a second matcher, so the gate and the reclaim cannot drift apart. Self-test check 14 drives it in both directions, against a decoy spawned after the reclaim, and asserts the decoy is not killed.- Every one of the six removed inline bodies was checked against the shared script: none differed in floor, ceiling, working directory or step environment beyond the intended changes, and the nightly Strix lane's job-level
TMPDIRstill reaches both reclaim and assert.
Deployment notes
- The first real run of the process gate will be on hardware. A leaked or slow-exiting engine process left over from an earlier run will now fail a lane that previously passed. Check the first few red preflights before reading them as regressions.
What this covered
Every file in the change, read completely:
.github/scripts/gpu-preflight.sh- the
scripts/reclaim-gpu.shdiff - all preflight and reclaim hunks in both workflows
- the
xtask/src/workflow_contract.rsdiff docs/testing.md
Plus the surrounding reclaim-gpu.sh functions, the e2e-selfhosted.yml path filter, and issues #446 and #294. Diff prw-base...HEAD = 21be003…63cacde0.
What was run:
bash scripts/reclaim-gpu.sh --self-test: 36 ok.cargo test -p xtask --bins -- workflow_contract::tests: 60 passed, all 10 relevant tests named, withpwshandtimeoutpresent.- shellcheck 0.11.0 with the repo profile: clean.
- Stubbed
rocm-smiand reclaim scenarios for every preflight branch. - Revert and per-branch mutation runs of the contract tests.
The two fan-out workers mutated the same checkout at the same time. One reverted the other's "default floor 16→1" mutant mid-run, so that single mutation result is unreliable; it concerns an unexercised default and is not used in any finding.
Did not run:
- Self-hosted GPU lanes: no hardware here, and the PR's run has not started.
- Windows-target compile of xtask.
- Prior-changes pass: review comments on earlier PRs to these files were not read.
Not reconciled against any discussion here.
`timeout` passes the command's own status through, so the preflight sees 127 when `reclaim-gpu.sh` is missing or unreadable and 2 when the two scripts disagree about a flag — deleting the `--assert-clear` dispatch arm produces exactly the 2. Both were reported as "a leaked E2E engine process is still holding it (listed above)", with nothing listed above: every lane fails, for a reason that is not true, naming a process that does not exist. Claim a survivor only for exit 1, the one code that means the scan ran and found somebody, and give every other non-zero exit its own message saying the host was shown nothing either way. Also correct the gate-ordering comment. It claimed "by here the memory is back, so anything the reclaim rules still match is a process that survived being reclaimed" — but the floor is 8/16 GiB against pools of ~62 and ~192 GiB, so on a healthy host it is met on the first poll with no wait at all. Clearing it therefore says nothing about whether a SIGKILLed process has finished exiting; what bounds that is the reclaim's own grace period. The ordering is still the right way round, it is simply not the proof of a settled card the comment claimed, and a process still dying when this runs is a false positive this gate can produce. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Each of these was mutation-tested and each stayed green on the mutation its
own message says it prevents.
The floor check was `step.contains("GPU_PREFLIGHT_MIN_FREE_GIB: 16")`. That
is a substring of `: 160`, and it searched the whole step, so commenting the
key out satisfied it too — on an APU lane that leaves the 8 GiB floor off
and the script's discrete 16 GiB default on, which is verbatim what the
assertion message says it stops. Parse the step's `env:` mapping and compare
the value exactly; `is_apu_preflight` reads the same parsed value, and its
two body spellings now match with comments stripped.
`every_native_linux_preflight_delegates_in_the_same_shape` compared only the
`run:` body, while its rationale named "a lane-local retry, a second floor"
as the drift it guards. `continue-on-error: true` on one lane turns a
required gate advisory, `if: false` removes it, and a lane-local
`GPU_PREFLIGHT_CEILING_SECS` is a second gate — none touch the body, all
three were green. Pin each step's keys to `name`/`env`/`run` and its step
environment to the floor alone, then compare the whole step around the body
across the six, so a lane cannot drift in either direction.
In `the_shared_preflight_asserts_no_leaked_engine_process_survives`,
`code.contains("--assert-clear")` could not fail. Filtering comment lines
fixed that once; the timeout fix then added a non-comment `echo "::error::…
(reclaim-gpu.sh --assert-clear) did not finish…"` and defeated the filter
again. Match the invocation itself — a non-comment line calling the script
with the flag — and require every such line to be the bounded form, so
neither deleting the call nor guarding it can stay quiet.
Add the three cases nothing reached. The real `reclaim-gpu.sh` is now
planted instead of the stub in one case, driven in both directions against a
decoy scoped by `RECLAIM_SELFTEST_SCOPE`: with the dispatch arm deleted the
harness always stubbed reclaim and the self-test called `assert_clear` as a
function, so nothing crossed the process boundary the lane crosses. Exit 2
and 127 are driven for the message they now get, and both
`--report-holders` call sites against a stub that sleeps past their bound.
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Thanks — all four blocking findings reproduced exactly as described. Fixed in Every mutation was re-run before and after, on the whole
On the vacuous assertion, confirmed directly rather than inferred: with the call line deleted and comment lines filtered, the one surviving non-comment line containing the flag is so What changed, briefly:
Devbox: Two observations, neither a change here:
|
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com> # Conflicts: # .github/workflows/nightly.yml # xtask/src/workflow_contract.rs
Closes #446. Advances #294.
Why the floor was insufficient
Every self-hosted lane gated on a fixed absolute free-memory floor applied to
whatever pool
rocm-smireports. What that detects is set by the pool'ssize, not by what a serve holds:
A leftover
llama-serverholding 10–20 GiB — the exact thing the step exists tocatch — clears all of them. The step only ever tripped near exhaustion.
What this adds
scripts/reclaim-gpu.sh --assert-clear: a new mode that runs the existingselect_leaked, exits non-zero naming the pid and command line of every leakedE2E engine process still running, and kills nothing. It reuses the rule, the
scope filter and the
E2E_ROOTS/ENGINE_MARKERSlists the reclaim alreadyacts on — no second matcher, which is how the original inline
pkill -fpatterns drifted out of agreement with what a serve looks like.
The preflight runs it after the memory floor is satisfied, deliberately: a
card still draining from the reclaim step's kills is the transient the bounded
wait exists to absorb, and asking first would report processes on their way out
as survivors. What that ordering does not buy, on these lanes, is proof the
drain has finished — per the table above the floor is met on the first poll on
any healthy host, so clearing it says nothing about whether a SIGKILLed process
has exited. What bounds that is the reclaim's own grace period. The ordering is
still the right way round; the script's comment now says only that, instead of
claiming a settled card it cannot establish.
The gate is bounded by
timeout 15and the outcomes are told apart: 0 is clear,1 is a named survivor, 124 is a scan that could not finish, and anything else is
a scan that could not be run at all (127 for a missing script, 2 for a flag the
two scripts disagree about). Each gets its own
::error::. A bound that passedon expiry, or a generic failure reported as "a leaked process is holding it
(listed above)" with nothing listed above, are both failures that point the next
reader at something that is not there.
What it deliberately does not replace
The memory floor stays. The two gates are complementary, not alternatives,
and each is the only one that catches its own case:
self-hosted host, which runs nothing a process check can match.
No proportional floor and no absolute-held check: both were considered in #446
and neither was chosen.
Landing it once instead of six times (#294)
The six native Linux preflight bodies were near-identical copies across two
files — which is why #442's fix had to land six times, and why the one time it
landed five nothing noticed for months. They are now one script,
.github/scripts/gpu-preflight.sh, invoked with a per-laneGPU_PREFLIGHT_MIN_FREE_GIB.The Strix/APU variant turned out to be a strict superset of the discrete one,
as its own comment claimed: a discrete card's VRAM total is far above any floor
so the carveout branch is unreachable, and the two-pool
--showmeminfo vram gttquery degrades to the single-pool call the discrete lanes always made. Verified
by test rather than assumed — including a new case for a discrete card on a
rocm-smithat rejects the two-pool form, the arm that actually has to fallback. So the only per-lane input is the floor. No lane's effective floor
changes: 16 GiB on the four discrete lanes, 8 GiB on the two Strix lanes,
asserted per lane by a contract test.
Everything load-bearing is preserved: the bounded wait,
timeout 15aroundevery
rocm-smi, retry-on-unparseable, the zero-total guards on both pools, thecarveout/GTT branch with its separate-query fallback, the one-shot memos
(
combined,gtt_absent,shown), deadline-checked-only-after-a-poll, andevery existing
::error::reason string.One subtlety worth flagging: as an inline step this ran under GitHub's
bash -e {0}, which has-ebut no pipefail, so the|| trueinfield()was insurance. As a standalone script it sets
-euo pipefail, which promotesthat
|| trueto load-bearing — the finalgrep -oEexits 1 for a pool thehost does not report, pipefail makes that the pipeline's status, and
-ewouldabort instead of leaving
gtt_totalempty for the carveout branch. Kept, withthe comment rewritten to say so.
Windows
The two PowerShell blocks stay separate copies (no bash to run the reclaim
with). Their comments no longer claim a sensitivity they lack and now state the
gap explicitly. Follow-up: a PowerShell equivalent reading
Win32_Process,the way the PowerShell reclaim step in the same job already does. The gap is
pinned by a test case rather than hidden — if those blocks ever grow an
equivalent, that case fails and names the reason.
The two WSL blocks also stay inline: they are advisory, warning where these
fail, which is a different contract rather than a copy.
Contract tests
xtask/src/workflow_contract.rsmatches on the inline YAML text, so extractionbroke several tests. They were re-pointed at what is now real, not weakened:
apu_lanes()reads the shared script from disk, so the gate/verdict testsstill drive the real body against fixture
rocm-smioutput.every_apu_preflight_block_measures_the_gtt_poolaccepts delegation, andseparately asserts the shared script queries both pools.
every_shell_preflight_polls_at_least_once_whatever_the_ceilingcounts the 6delegating lanes and the 2 remaining inline bodies, and drives the shared
script.
apu_preflight_twins_do_not_driftis unchanged but documented: it stillcarries the PowerShell and WSL pairs; the native pair is now trivially equal.
every_native_linux_preflight_delegates_to_the_shared_script— theanti-drift guarantee the equality check can no longer provide, plus each lane
must name its own floor rather than inherit the default.
the_shared_preflight_asserts_no_leaked_engine_process_survives— pins both gates, matching the
--assert-clearinvocation rather than anymention of the flag, and requiring it to be the
timeout 15-bounded form with124 and 1 handled apart.
preflight_runs_the_real_reclaim_assert_clear— plants the realscripts/reclaim-gpu.shinstead of the stub and drives the gate in bothdirections against a decoy scoped by
RECLAIM_SELFTEST_SCOPE. Everything elsestubs reclaim and the script's own self-test calls
assert_clearas a bashfunction, so until this nothing crossed the process boundary the lane crosses.
preflight_distinguishes_a_scan_that_never_ran_from_a_survivor—drives reclaim exiting 2 and 127.
preflight_bounds_the_holder_report— drives both--report-holderscall sites against a stub that sleeps past their bound.
Verification
Run on the Linux devbox at the pushed SHA (confirmed against the remote
git rev-parse HEAD, not a stale worktree):bash scripts/reclaim-gpu.sh --self-test— all--assert-clearchecks pass.cargo test -p xtask— 259 passed, 0 failed, withpwshand GNUtimeoutpresent so the PowerShell and bounded-scan cases actually ran.
cargo clippy --workspace --all-targets -- -D warnings— clean.shellcheck --enable=all --exclude=SC2310,SC2312— clean on both scripts.bash -non both.Falsified, not just exercised — a check that cannot fail is worthless. Every
line below is a mutation applied to the tree, with the whole
workflow_contractmodule run before and after the test fixes:16→160…delegates_to_the_shared_script…delegates_to_the_shared_script+ 3 morecontinue-on-error: trueon one lane…delegates_in_the_same_shapeif: falseon one lane…delegates_in_the_same_shapeGPU_PREFLIGHT_CEILING_SECS…delegates_in_the_same_shape--assert-clearcall line deleted…asserts_no_leaked_engine_process_survives, on "no longer RUNS"if false; then … fi--assert-clear)arm deleted fromreclaim-gpu.sh'smainpreflight_runs_the_real_reclaim_assert_cleartimeout 15removed from the gate-2 holder reportpreflight_bounds_the_holder_reporttimeout 15removed from the ceiling holder reportpreflight_bounds_the_holder_reportpreflight_distinguishes_a_scan_that_never_ran_from_a_survivorThe first assertion of the shape test was vacuous in a way worth spelling out,
because it had already been repaired once: it was matching the script's own
prose about
--assert-clear, was fixed in1872c76eby filtering commentlines, and was then defeated again by
1cde744, which added a non-commentecho "::error::… (reclaim-gpu.sh --assert-clear) did not finish…". With thecall deleted, that echo alone still satisfied
code.contains("--assert-clear").It now matches the invocation itself.
End-to-end against a live decoy, through the preflight rather than beside it: a
llama-serverunder an E2E root, scoped byRECLAIM_SELFTEST_SCOPE, givesexit=1naming the pid; once reaped,exit=0.Not verified: the self-hosted GPU lanes themselves.
E2E self-hostedrun38563490556 for this head has been
pendingwith zero jobs since it wascreated — the repo's self-hosted queue is 78 runs deep, so this is the backlog
rather than anything about this branch. Four of the six callers are
nightly.ymllanes that never run on a PR in any case. The preflight's behaviour against real
rocm-smioutput is covered only by the fixture-driven contract tests, and thefirst real exercise of the process gate will be the first self-hosted run.