Skip to content

ci(e2e): GPU preflight asserts no leaked engine process survives - #672

Merged
r0x0r merged 10 commits into
mainfrom
ci/rocmai-446-gpu-preflight-process-check
Oct 15, 2026
Merged

r0x0r merged 10 commits into
mainfrom
ci/rocmai-446-gpu-preflight-process-check

Conversation

@r0x0r

@r0x0r r0x0r commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #446. Advances #294.

Why the floor was insufficient

Every self-hosted lane gated on a fixed absolute free-memory floor applied to
whatever pool rocm-smi reports. What that detects is set by the pool's
size
, not by what a serve holds:

lane pool floor trips only when held exceeds
Strix Halo (GTT aperture) ~62 GiB 8 GiB ~54 GiB
MI300X 192 GiB 16 GiB ~176 GiB

A leftover llama-server holding 10–20 GiB — the exact thing the step exists to
catch — clears all of them. The step only ever tripped near exhaustion.

What this adds

scripts/reclaim-gpu.sh --assert-clear: a new mode that runs the existing
select_leaked, exits non-zero naming the pid and command line of every leaked
E2E engine process still running, and kills nothing. It reuses the rule, the
scope filter and the E2E_ROOTS / ENGINE_MARKERS lists the reclaim already
acts on — no second matcher, which is how the original inline pkill -f
patterns drifted out of agreement with what a serve looks like.

The preflight runs it after the memory floor is satisfied, deliberately: a
card still draining from the reclaim step's kills is the transient the bounded
wait exists to absorb, and asking first would report processes on their way out
as survivors. What that ordering does not buy, on these lanes, is proof the
drain has finished — per the table above the floor is met on the first poll on
any healthy host, so clearing it says nothing about whether a SIGKILLed process
has exited. What bounds that is the reclaim's own grace period. The ordering is
still the right way round; the script's comment now says only that, instead of
claiming a settled card it cannot establish.

The gate is bounded by timeout 15 and the outcomes are told apart: 0 is clear,
1 is a named survivor, 124 is a scan that could not finish, and anything else is
a scan that could not be run at all (127 for a missing script, 2 for a flag the
two scripts disagree about). Each gets its own ::error::. A bound that passed
on expiry, or a generic failure reported as "a leaked process is holding it
(listed above)" with nothing listed above, are both failures that point the next
reader at something that is not there.

What it deliberately does not replace

The memory floor stays. The two gates are complementary, not alternatives,
and each is the only one that catches its own case:

  • the process check catches the 10–20 GiB leftover serve the floor misses;
  • the floor catches genuine exhaustion by a non-E2E tenant on a shared
    self-hosted host, which runs nothing a process check can match.

No proportional floor and no absolute-held check: both were considered in #446
and neither was chosen.

Landing it once instead of six times (#294)

The six native Linux preflight bodies were near-identical copies across two
files — which is why #442's fix had to land six times, and why the one time it
landed five nothing noticed for months. They are now one script,
.github/scripts/gpu-preflight.sh, invoked with a per-lane
GPU_PREFLIGHT_MIN_FREE_GIB.

The Strix/APU variant turned out to be a strict superset of the discrete one,
as its own comment claimed: a discrete card's VRAM total is far above any floor
so the carveout branch is unreachable, and the two-pool --showmeminfo vram gtt
query degrades to the single-pool call the discrete lanes always made. Verified
by test rather than assumed — including a new case for a discrete card on a
rocm-smi that rejects the two-pool form, the arm that actually has to fall
back. So the only per-lane input is the floor. No lane's effective floor
changes
: 16 GiB on the four discrete lanes, 8 GiB on the two Strix lanes,
asserted per lane by a contract test.

Everything load-bearing is preserved: the bounded wait, timeout 15 around
every rocm-smi, retry-on-unparseable, the zero-total guards on both pools, the
carveout/GTT branch with its separate-query fallback, the one-shot memos
(combined, gtt_absent, shown), deadline-checked-only-after-a-poll, and
every existing ::error:: reason string.

One subtlety worth flagging: as an inline step this ran under GitHub's
bash -e {0}, which has -e but no pipefail, so the || true in field()
was insurance. As a standalone script it sets -euo pipefail, which promotes
that || true to load-bearing
— the final grep -oE exits 1 for a pool the
host does not report, pipefail makes that the pipeline's status, and -e would
abort instead of leaving gtt_total empty for the carveout branch. Kept, with
the comment rewritten to say so.

Windows

The two PowerShell blocks stay separate copies (no bash to run the reclaim
with). Their comments no longer claim a sensitivity they lack and now state the
gap explicitly. Follow-up: a PowerShell equivalent reading Win32_Process,
the way the PowerShell reclaim step in the same job already does. The gap is
pinned by a test case rather than hidden — if those blocks ever grow an
equivalent, that case fails and names the reason.

The two WSL blocks also stay inline: they are advisory, warning where these
fail, which is a different contract rather than a copy.

Contract tests

xtask/src/workflow_contract.rs matches on the inline YAML text, so extraction
broke several tests. They were re-pointed at what is now real, not weakened:

  • apu_lanes() reads the shared script from disk, so the gate/verdict tests
    still drive the real body against fixture rocm-smi output.
  • every_apu_preflight_block_measures_the_gtt_pool accepts delegation, and
    separately asserts the shared script queries both pools.
  • every_shell_preflight_polls_at_least_once_whatever_the_ceiling counts the 6
    delegating lanes and the 2 remaining inline bodies, and drives the shared
    script.
  • apu_preflight_twins_do_not_drift is unchanged but documented: it still
    carries the PowerShell and WSL pairs; the native pair is now trivially equal.
  • New every_native_linux_preflight_delegates_to_the_shared_script — the
    anti-drift guarantee the equality check can no longer provide, plus each lane
    must name its own floor rather than inherit the default.
  • New the_shared_preflight_asserts_no_leaked_engine_process_survives
    — pins both gates, matching the --assert-clear invocation rather than any
    mention of the flag, and requiring it to be the timeout 15-bounded form with
    124 and 1 handled apart.
  • New preflight_runs_the_real_reclaim_assert_clear — plants the real
    scripts/reclaim-gpu.sh instead of the stub and drives the gate in both
    directions against a decoy scoped by RECLAIM_SELFTEST_SCOPE. Everything else
    stubs reclaim and the script's own self-test calls assert_clear as a bash
    function, so until this nothing crossed the process boundary the lane crosses.
  • New preflight_distinguishes_a_scan_that_never_ran_from_a_survivor —
    drives reclaim exiting 2 and 127.
  • New preflight_bounds_the_holder_report — drives both --report-holders
    call sites against a stub that sleeps past their bound.

Verification

Run on the Linux devbox at the pushed SHA (confirmed against the remote
git rev-parse HEAD, not a stale worktree):

  • bash scripts/reclaim-gpu.sh --self-test — all --assert-clear checks pass.
  • cargo test -p xtask — 259 passed, 0 failed, with pwsh and GNU timeout
    present so the PowerShell and bounded-scan cases actually ran.
  • cargo clippy --workspace --all-targets -- -D warnings — clean.
  • shellcheck --enable=all --exclude=SC2310,SC2312 — clean on both scripts.
    bash -n on both.

Falsified, not just exercised — a check that cannot fail is worthless. Every
line below is a mutation applied to the tree, with the whole
workflow_contract module run before and after the test fixes:

mutation before after, caught by
a discrete lane's floor 16 → 160 green …delegates_to_the_shared_script
an APU lane's floor key commented out green …delegates_to_the_shared_script + 3 more
continue-on-error: true on one lane green …delegates_in_the_same_shape
if: false on one lane green …delegates_in_the_same_shape
a lane-local GPU_PREFLIGHT_CEILING_SECS green …delegates_in_the_same_shape
the --assert-clear call line deleted red, on the wrong assertion …asserts_no_leaked_engine_process_survives, on "no longer RUNS"
that call wrapped in if false; then … fi red, on the wrong assertion same test, on "does not invoke … as a plain, bounded command"
--assert-clear) arm deleted from reclaim-gpu.sh's main green (suite and self-test) preflight_runs_the_real_reclaim_assert_clear
timeout 15 removed from the gate-2 holder report green preflight_bounds_the_holder_report
timeout 15 removed from the ceiling holder report green preflight_bounds_the_holder_report
every non-124 failure reported as a survivor n/a (new branch) preflight_distinguishes_a_scan_that_never_ran_from_a_survivor

The first assertion of the shape test was vacuous in a way worth spelling out,
because it had already been repaired once: it was matching the script's own
prose about --assert-clear, was fixed in 1872c76e by filtering comment
lines, and was then defeated again by 1cde744, which added a non-comment
echo "::error::… (reclaim-gpu.sh --assert-clear) did not finish…". With the
call deleted, that echo alone still satisfied code.contains("--assert-clear").
It now matches the invocation itself.

End-to-end against a live decoy, through the preflight rather than beside it: a
llama-server under an E2E root, scoped by RECLAIM_SELFTEST_SCOPE, gives
exit=1 naming the pid; once reaped, exit=0.

Not verified: the self-hosted GPU lanes themselves. E2E self-hosted run
38563490556 for this head has been pending with zero jobs since it was
created — the repo's self-hosted queue is 78 runs deep, so this is the backlog
rather than anything about this branch. Four of the six callers are nightly.yml
lanes that never run on a PR in any case. The preflight's behaviour against real
rocm-smi output is covered only by the fixture-driven contract tests, and the
first real exercise of the process gate will be the first self-hosted run.

r0x0r and others added 3 commits October 14, 2026 11:29
The GPU preflight's free-memory floor is a fixed 8 GiB (Strix) or 16 GiB
(discrete) applied to whatever pool rocm-smi reports, so what it detects is
set by the pool's SIZE and not by what a serve holds: against a ~62 GiB Strix
GTT aperture it trips only past ~54 GiB held, and against a 192 GiB MI300X
only past ~176 GiB. A leftover llama-server holding 10-20 GiB — the exact
thing the step exists to catch — clears every lane.

Add `reclaim-gpu.sh --assert-clear`: reusing select_leaked, it exits non-zero
naming any leaked E2E engine process still running, and kills nothing. The
preflight runs it after the memory floor is satisfied, so a card still
draining from the reclaim step's kills is absorbed by the bounded wait rather
than reported.

Keep the floor. The two gates are complementary, not alternatives: the
process check catches the leftover serve the floor misses, and the floor
catches a non-E2E tenant exhausting a shared host, which no process check can
see.

Extract the six near-identical native Linux preflight bodies into one
.github/scripts/gpu-preflight.sh, so this landed once rather than six times
(advances #294). The Strix/APU variant was a strict superset of the discrete
one — the two-pool query degrades to the single-pool one, and a discrete
card's VRAM total never reaches the carveout branch — so the only per-lane
input is GPU_PREFLIGHT_MIN_FREE_GIB. No lane's effective floor changes.

The two PowerShell blocks keep memory-only semantics and say so; a
Win32_Process equivalent is follow-up.

Closes: #446
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
The extraction in the previous commit rests on the APU preflight body being a
strict superset of the discrete one: a discrete card's VRAM total is far above
any floor so the carveout branch is unreachable, and the two-pool `--showmeminfo
vram gtt` query degrades to the single-pool call the discrete lanes always made.

The existing cases cover a rocm-smi that ACCEPTS the two-pool form on a discrete
card, and one that rejects it on a carveout. Nothing covered a rejection on a
full-size pool — the arm that actually has to fall back on an Instinct host with
an older rocm-smi.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…prose

The shape assertion searched the whole script for `--assert-clear`, which the
header comment and the error message both name. Replacing the
`if bash "${RECLAIM_SCRIPT}" --assert-clear; then` condition with `if true; then`
left the test green — it was matching the explanation of the gate rather than
the gate. Filter comment lines out first.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r
r0x0r requested a review from a team as a code owner October 14, 2026 08:46
@r0x0r
r0x0r requested a review from volen-silo October 14, 2026 08:46
The two WSL bodies keep only the free-memory floor: they do not run
`reclaim-gpu.sh --assert-clear`, because they execute inside the WSL
distro through Invoke-WslBash.ps1, where the repo checkout is not at a
path the step can assume.

Their comment still claimed a leftover serve "still fails", which is the
overclaim #446 exists to correct and which the native lanes now actually
deliver. State the weaker guarantee and why, rather than leaving the
native and WSL wording identical when the gates no longer are.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up in 492c4e73: corrected the two WSL advisory preflight comments, which still claimed a leftover serve "still fails".

That wording was accurate before this PR, when every lane had the same (coarse) gate. It is not accurate now: the six native Linux lanes gained reclaim-gpu.sh --assert-clear, and the WSL bodies did not — they run inside the WSL distro via Invoke-WslBash.ps1, where the repo checkout is not at a path the step can assume. So on those two lanes a serve holding a few GiB of a whole GTT aperture still clears the floor, which is exactly the #446 overclaim.

Leaving the native and WSL comments identical while the gates diverged would have re-created the defect this PR fixes, one layer down. The comments now state the weaker guarantee and why, and point at the Windows gap as the companion follow-up.

Verified on Linux at 492c4e73: cargo test -p xtask exit 0, 254 passed / 0 failed (the contract tests match on this YAML text, so the comment edit had to be checked against them), and reclaim-gpu.sh --self-test exit 0 including the new --assert-clear cases.

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work (minor)

Full review of the whole change. Implements the process-check gate from ROCMAI-446 / #446 in full; separately advances #294 (preflight-body deduplication) without closing it — the PR body scopes this accurately as "advances," not "closes."

Reviewed via 4 independent file-group passes (core scripts; workflow YAML; contract tests + docs; agent-instruction/history/comments) plus a consolidation pass and a targeted design-review check on self-hosted runner concurrency. Diff range dfe1df3aff0974cc58ef2be5cf5b6a276fbd98d3...492c4e73362e8d45d6e511e9cad232c38c67a367 (PR head at review time). All 25 CI checks were green at review time.

Blocking (minor, no functional consequence)

  • New doc prose cites a tracked file as a bare backtick path instead of a relative markdown link — docs/testing.md:1192
    AGENTS.md §5 asks for file citations added/edited in markdown to be relative links rather than bare backtick paths, so a future rename/move doesn't go stale silently. This PR's new paragraph cites the new .github/scripts/gpu-preflight.sh as `.github/scripts/gpu-preflight.sh` rather than as a link. Mitigating: the entire file already cites every other file this same bare way (12+ pre-existing instances — zero markdown-style links anywhere in docs/testing.md), so this continues an established pattern in this doc rather than introducing a new one, and CI's lychee job has nothing to check either way. Fix: wrap the new citation in a relative link, e.g. [`gpu-preflight.sh`](../.github/scripts/gpu-preflight.sh); or, if bare-citation style is intentional for this doc, worth noting as an explicit exception.

Non-blocking / for your consideration

  • Escalating the leaked-process check to a hard CI-failing assertion inherits select_leaked's existing unanchored-substring over-match exposure at a higher cost. Previously a false match meant an extra kill; now it fails the whole job. The code's own comment already acknowledges this tradeoff directly, so flagging only so it's a conscious sign-off rather than implicit.
  • The real discrete-lane 16 GiB floor is asserted present in the workflow YAML only by a static string check; no dynamic GateCase exercises a near-16-GiB-boundary scenario. All current discrete fixtures have huge margins, so the floor's specific value never changes a verdict today. A boundary case (~10–15 GiB free) would close the gap.
  • No contract test pins that all six native-Linux call sites invoke the shared script identically, the way the PowerShell/WSL twins are pinned. Low risk today (each site is a one-line delegation), but there's no structural guarantee against future re-inlining drift.
  • Self-hosted runner concurrency was checked directly: each GPU lane targets a distinct hardware runs-on label, and a GitHub self-hosted runner executes one job at a time, so the new process gate can't race against a concurrent job on the same host within this workflow. No finding here — noted since it was the main open design question worth checking.

Positive signals

  • assert_clear() deliberately reuses select_leaked() rather than re-implementing matching — avoiding exactly the kind of matcher drift that caused the original six inline pkill -f copies to diverge from their PowerShell mirrors.
  • The new contract test the_shared_preflight_asserts_no_leaked_engine_process_survives strips comment lines before matching the gate, with a comment documenting it was tightened after an earlier version matched the script's own prose about the gate rather than the gate itself — independently reproduced via the same mutation during this review.
  • field()'s || true correctly flips from cosmetic insurance (inline step, no pipefail) to load-bearing (standalone script, set -euo pipefail) — a subtle shell-semantics trap, called out explicitly in a comment rather than silently inherited.

Deployment notes

A leaked-process false positive (see non-blocking item above) now fails a self-hosted CI job outright instead of being silently reclaimed — worth a heads-up to whoever triages self-hosted lane failures, since the failure signature changes.


This is a feedback-only review comment and does not block the merge.

The existing delegation test asks whether each lane MENTIONS the shared
script. It cannot see a lane that keeps the delegation and grows a line of
its own above it — one more `pkill -f`, a lane-local retry — which is how
the original six inline copies drifted from their PowerShell mirrors in the
first place.

Compare the six bodies to each other, as the PowerShell/WSL twins already
are, then pin the shape they agree on: a lone `bash <script>` line.
Equality alone passes six identically re-inlined copies; the shape check
alone misses five of six.

Also drive the discrete lanes' 16 GiB floor across its own boundary. Every
other discrete fixture clears or misses it by over a hundred GiB, so the
number never decided a verdict; at 15 GiB free it must fail and at exactly
16 it must pass.

And cite the new script as a relative link rather than a bare backtick path
(AGENTS.md §5), so lychee catches a rename.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — this was a useful review. Addressed in b65b6972 (492c4e73..b65b6972). Point by point.

1. Bare backtick path (blocking, minor) — fixed

docs/testing.md:1192 now reads [`gpu-preflight.sh`](../.github/scripts/gpu-preflight.sh), the form you suggested. Verified it resolves: docs/ → repo root → .github/scripts/gpu-preflight.sh, a tracked file; lychee is offline = true and exclude_path only covers docs/rocm-docs/, so docs-links really checks it. The paragraph is reflowed around the longer line.

I took the link rather than declaring an exception. AGENTS.md §5 ends with "This covers citations you add or edit, not converting existing bare paths in an unrelated change" — which both obliges the new citation and scopes the fix, so the other 12 pre-existing bare paths are deliberately left alone. This one points at a file the PR creates, which is where a silent rot costs the most.

2. Nothing pinned that the six call sites delegate identically — fixed, and the gap was real

New every_native_linux_preflight_delegates_in_the_same_shape. The existing test asks whether each lane mentions the script; the new one compares the six run: bodies to each other (via first_line_difference, as the PowerShell/WSL twins are compared) and then pins the shape they agree on: a lone bash <script> line. Equality alone would pass six identically re-inlined copies; the shape check alone would miss five of six. The per-lane floor is deliberately excluded — it rides in env:, not the body, which is why the bodies can be identical.

Mutation evidence (all on the devbox, b65b6972):

mutation old test new test
one lane grows pkill -f llama-server || true above its delegation ok FAILED
all six reshaped to bash ./.github/... (still identical to each other) ok FAILED

The first is the decisive one: the pre-existing test stays green against exactly the re-inlining drift you flagged. The new failure names it —

a lane has grown a body of its own again ... first difference at body line 1:
  e2e-selfhosted.yml GPU preflight #0: "bash .github/scripts/gpu-preflight.sh"
  nightly.yml GPU preflight #0: "pkill -f llama-server || true"

3. No near-16-GiB GateCase — added, and it was cheap

Two cases, because run_preflight already documents that a case wanting the discrete floor overrides GPU_PREFLIGHT_MIN_FREE_GIB through env:

  • discrete card one GiB under the 16 GiB floor — 192 GiB total / 177 used → 15 GiB free, must fail.
  • discrete card exactly at the 16 GiB floor — 176 used → 16 GiB free, must pass (pins -ge, not -gt).

Mutation evidence, and this is the part worth reading:

  • Hardcoding the floor to 8 GiB (i.e. the discrete lanes' 16 silently gone) — the entire 255-test xtask suite passed with the two new cases removed. With them, it fails on a self-contradicting log: GPU ready: 15 GiB VRAM free (>= 16 GiB). Your "the floor's specific value never changes a verdict today" was exactly right; it does now.
  • -ge → -gt — fails on waiting: 16 GiB VRAM free (< 16 GiB). No prior fixture lands on the number, so nothing else saw it.

Both lanes and the Windows block are driven; pwsh 7.4.6 was installed on the devbox so the PowerShell cases really executed rather than skipped (zero skipping the Windows lane in the output).

4. Unanchored over-match now fails a job — deliberate, signing off explicitly

Confirmed as a conscious decision, not inherited. The reasoning: a false positive that fails loudly is strictly preferable to a leaked serve that silently corrupts a later run's results. The failure mode of the old behaviour (an extra kill) was cheap but so was its detection value; #446 exists because the quiet path won for months. The matcher's exposure is unchanged — assert_clear() reuses select_leaked() rather than re-implementing it, which is what keeps the cost of a future tightening at one site.

Taking your deployment note: the failure signature for self-hosted lane triage has changed, and the error names it — a leaked E2E engine process is still holding it (listed above) followed by --report-holders output, distinct from the memory-floor timeout. I have not added a separate heads-up doc; if you would rather that live somewhere durable than in this thread, say where and I will add it.

Verification

On the devbox (macOS cannot build this repo), at b65b6972:

  • bash scripts/reclaim-gpu.sh --self-test — all checks passed, including all four --assert-clear cases
  • bash -n + shellcheck 0.10.0 on both scripts — clean
  • cargo test -p xtask — 255 passed, 0 failed
  • cargo clippy --workspace --all-targets -- -D warnings — 0

As the PR description already says, the self-hosted GPU lanes themselves cannot be exercised from here.

On the positive finding

Noted, and genuinely reassuring that you reproduced the comment-stripping mutation independently — that is the difference between the test being fix-differentiating and merely passing. It was caught the same way, after an earlier version matched the script's own prose about --assert-clear instead of the gate, which is why every assertion added here was mutated before being proposed.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed this (closes #446, partially advances #294) with a correctness pass plus a separate Standards/Spec pass. CI is fully green, and the core approach - a second gate asserting no leaked E2E engine process survives, layered after the existing free-memory floor - is sound and well-reasoned; I traced the two-gate ordering, the set -euo pipefail semantics change, the xtask contract-test rewrites, and the new reclaim-gpu.sh --self-test case, and all of it holds up.

Requesting changes on one concrete gap, left as an inline comment below: every rocm-smi call in the new gpu-preflight.sh is wrapped in timeout 15 specifically because the driver can hang, but the new --assert-clear call that implements this PR's own headline gate has no such bound. If the process scan ever stalls, this step has no internal ceiling and rides to the job's own timeout - exactly the failure mode the rest of the script exists to avoid.

Also worth noting, not blocking:

  1. One structural fact rather than a defect: the self-hosted GPU lanes that actually exercise --assert-clear aren't part of per-PR CI at all, so this PR's own run validates the new behavior only indirectly (xtask contract tests + the bash --self-test harness), never on live hardware - matches the PR description's own disclosure.
  2. Against #294 specifically: the duplicated reclaim-gpu.sh invocation across the 8 jobs is untouched by this PR (only the shared preflight block was deduplicated), and #294's acceptance criterion for a lane-count-vs-docs contract test isn't implemented here (docs/testing.md is touched, docs/ci-hardware-testing.md is not). Also, this PR uses a shared shell script rather than #294's proposed composite GitHub Action - a reasonable deviation, just flagging it as one.

None of the non-blocking items need to hold this up; the timeout gap does, since it's a real hang risk in the gate this PR adds.

# and self-tests them against decoy processes. A second matcher in CI is how
# the original `pkill -f` patterns drifted.
if bash "${RECLAIM_SCRIPT}" --assert-clear; then
exit 0

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every other external call in this script (rocm-smi, 4 call sites above) is wrapped in timeout 15 because the driver can hang. This call - the new assert-clear gate this PR adds - isn't. If reclaim-gpu.sh --assert-clear's process scan ever stalls (e.g. a wedged ps/driver query inside it), this step has no bound and rides to the job's own timeout instead of failing fast the way the rest of this script is designed to. Worth wrapping in timeout <n> the same as the rocm-smi calls above, with a clear error message on the timeout-exit-code (124) path.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1cde744 (+ test in 63cacde) — you were right that this was more than a style inconsistency, so the comment now says why.

What is bounded. timeout 15 bash "${RECLAIM_SCRIPT}" --assert-clear, same style and same value as the four rocm-smi calls above. The two --report-holders diagnostics got the same bound: having just established that the scan can stall, an unbounded diagnostic on the failure path would spend the ceiling the bound just saved.

What a timeout now does, and why that direction. It fails the gate, with its own message, as a third outcome distinct from clear/survivor:

::error::GPU preflight failed: the leaked-process scan (reclaim-gpu.sh --assert-clear) did not
finish within 15s and was timed out — the host could NOT be shown clear. …

A scan that could not complete is not evidence the host is clean, so passing on it would convert a hang into a silent green on exactly the hosts #446 is about. The status is captured (|| assert_rc=$?) rather than tested with if, so under set -euo pipefail the 124 path is reached and named rather than collapsing into the survivor message.

The reason the bound matters more here than for rocm-smi is now in the script: select_leaked walks /proc/[0-9]*/cmdline and reads each entry, and a process wedged in uninterruptible sleep can block that read — which is the state a wedged GPU process is in. The hang risk is correlated with the condition being detected, not independent of it.

Evidence the bound actually fires (devbox, real gpu-preflight.sh, --assert-clear stubbed to exec sleep 60, fixture rocm-smi showing a free aperture):

GPU ready: 61 GiB GTT free (>= 8 GiB).
STUB reclaim-gpu.sh called: --assert-clear
::error::GPU preflight failed: the leaked-process scan … did not finish within 15s …
STUB reclaim-gpu.sh called: --report-holders
--- exit=1 elapsed=15s ---

Control, same stub against b65b697: --- exit=0 elapsed=60s ---. Unbounded it did not merely hang — it rode the stall out and then passed the gate. Clear host still exits 0 in 0s; a named survivor still exits 1 with the unchanged #446 message.

Pinned, not just measured once. preflight_bounds_the_leaked_process_scan drives the real script against a never-returning scan and asserts exit 1, the timeout wording, that holders are still reported, and that it ends at the bound (the stub sleeps 60s, so elapsed separates "cut short" from "happened to return"). The shape test gained the complementary pins (timeout 15 … present; 124 handled distinctly). Both mutations are caught: dropping the timeout fails both tests after 60s; changing -eq 124 to -eq 125 fails both at 15s with the "did not say so" message.

I did not add a reclaim-gpu.sh --self-test case: the bound lives in gpu-preflight.sh, not in the reclaim, and the self-test's job is the matching rules. The xtask harness is where the preflight's --assert-clear behaviours are already pinned, and it was free — cargo test -p xtask is 20.09s without the new case and 20.10s with it, since it runs in parallel with the existing multi-second cases.

devbox: 256 xtask tests pass (255 + the new one), reclaim-gpu.sh --self-test green including all four --assert-clear cases, cargo clippy --workspace --all-targets -D warnings clean, bash -n + shellcheck clean. The self-hosted GPU lanes still cannot be exercised from here, as the PR body discloses.

Your two non-blocking items are deliberately left as they are: (1) the live-hardware validation gap — this round adds one more off-hardware proof but does not close it; and (2) the #294 deltas (duplicated reclaim-gpu.sh invocation untouched, no lane-count-vs-docs contract test, docs/ci-hardware-testing.md untouched, shared script rather than a composite action). The PR body still says "advances" rather than "closes" for #294.

r0x0r added 2 commits October 14, 2026 15:09
Every rocm-smi call in gpu-preflight.sh runs under `timeout 15` because the
driver can hang. The `--assert-clear` gate this PR adds had no such bound, so
the one step added to stop a job hanging to its cap could itself hang to it.

The risk is not hypothetical and not independent of what is being detected:
`select_leaked` reads /proc/<pid>/cmdline for every process, and a holder
wedged in uninterruptible sleep can block that read — which is the state a
wedged GPU process is in, i.e. exactly the condition the gate fires on.

The bound FAILS the gate. `timeout` reports expiry as 124, which is handled as
its own outcome with its own message: a scan that could not finish is not
evidence the host is clear, and reporting it as either a pass or a named
survivor misdescribes it. The `--report-holders` diagnostics are bounded the
same way — having just established the scan can stall, an unbounded diagnostic
would spend the ceiling the bound just saved.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
A timeout nobody has exercised is a timeout nobody knows works, and this one
can never be exercised from per-PR CI on live hardware: the lanes that run
`--assert-clear` for real are self-hosted and nightly.

So the reclaim stub gains a mode that never returns, and the native preflight
is run against it: the run must fail, say the scan did not finish within 15s,
still name what it could see holding the card, and END AT THE BOUND. The stub
sleeps 60s, so the elapsed time is what separates "the bound cut it short"
from "the scan happened to return" — unbounded, that case takes a full minute
and then PASSES the gate, because the stub exits 0.

`exec sleep`, not a plain one: the bound's SIGTERM has to land on the sleep
itself, or it is orphaned still holding the captured stdout pipe and the
harness waits out the stub rather than timing the script.

The shape test gains the complementary pin — the call is wrapped in
`timeout 15` and 124 is handled distinctly — so removing either is named
immediately instead of by a case that takes a minute to go red.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

@tomastola tomastola left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 63cacde

This is an approval filed at the explicit request of the engineer driving this review, overriding the automation's usual rule of never filing one. The review itself found nothing blocking.

Review — no blocking findings

Full review of the whole change, read at 63cacde0: the original 6-file change reviewed at 492c4e73, plus the three commits that landed after it (b65b6972, 1cde744, 63cacde0), which were reviewed in full.
The description names Closes #446. Advances #294. The change does what those two issues ask: #446 option 3, a process-based assertion with the floor kept, and #294 for the six native Linux preflight copies. The description predates the three later commits and does not mention them.

Blocking

None

Non-blocking

None

Decisions for the author

  • The description does not cover the three commits added after it — non-blocking-improvement
    1cde744 adds a behaviour change: gpu-preflight.sh now bounds --assert-clear and --report-holders with timeout 15, and fails with its own message when the scan times out (exit 124). The description's "What this adds" and "Everything load-bearing is preserved" sections, and its verification numbers (cargo test -p xtask — 254 passed), predate it. The docs were updated and the behaviour is pinned by tests; only the description is stale. Add a line for it.
  • --assert-clear turns the reclaim's substring over-match into a red lane instead of a kill — tradeoff
    The assertion shares select_leaked with the reclaim, which is the benefit: no second matcher to drift. The cost is that any process whose command line contains an ENGINE_MARKERS or E2E_ROOTS substring now fails the lane. Only $$/PPID are excluded. Confirm a failure is wanted over a warning; I think it is, because that is the point of the gate. Confidence 40.
  • Concurrent matrix legs on one host now fail visibly instead of being killed silently — tradeoff
    nightly.yml e2e-gpu-nightly-strix has matrix: channel: [release, nightly]. If two legs can share a host, one leg's live serve already got SIGTERMed by the other's reclaim, and can now also fail the other's preflight. The exposure is not new; only the failure mode changed. I could not establish whether they share a host.
  • The 15s bound relies on timeout killing the scan — non-blocking-improvement
    timeout 15 sends SIGTERM only, with no -k. A process truly wedged in uninterruptible sleep ignores SIGTERM until its syscall returns, so the bound is firm for killable waits and best-effort for a hard D-state hang. Adding -k 5 would make the escalation explicit. The comment claims the ceiling for the whole class. Not verified on real hardware. Confidence 30.
  • The native-lane floor sits in step env:, outside what the drift test compares — tradeoff
    Swapping the floor between two lanes of the same class is not caught. The new every_native_linux_preflight_delegates_in_the_same_shape test compares the bodies to each other and pins the lone-delegation shape, and it states plainly that the floor is deliberately not part of it. The per-lane value is still pinned by every_native_linux_preflight_delegates_to_the_shared_script. Confidence 30.
  • Gate 2 checks once, with no retry — non-blocking-improvement
    After the floor is met, --assert-clear runs a single time. The reclaim step SIGKILLs and does not wait for exit, so a process still dying could fail the lane. A separate step sits between them and process_alive treats zombies as dead, so exposure is small. Confidence 20.
  • Multi-GPU hosts: gate 1 reads only the last GPU's row — non-blocking-improvement
    field ends in tail -1. Pre-existing and documented in a comment; gate 2 is process-based and has no such blind spot.

Positive signals

  • The new timeout is tested by running it. preflight_bounds_the_leaked_process_scan drives a stub that sleeps 60s against the 15s bound and asserts the exit code, the specific message and the elapsed time. Using exec sleep so the signal lands on the sleep itself rather than an orphan holding the pipe is the detail that makes the test time the bound and not the stub.
  • The 16 GiB floor is now driven across its own boundary. The 15 GiB and exactly-16 GiB cases pin both the value and >= against >. Previously every discrete fixture cleared or missed the floor by over a hundred GiB, so the number decided nothing.
  • A timeout fails the gate with its own reason. A scan that could not finish is not treated as a clear host, and the exit 124 is captured rather than tested with if, so clear, survivor and never-finished stay distinguishable.
  • The anti-drift test compares the six bodies to each other and also pins the shape. Either half alone would be satisfied by six identical re-inlined copies or would miss one of six.

Deployment notes

  • No self-hosted GPU lane has run against this head. The first real run of the process gate and of the new 15s bound will be the first self-hosted run after merge, against real rocm-smi output and a real /proc. Watch that run.
  • The assertion can newly fail lanes on hosts with a leaked or over-matched engine process. That is the intended behaviour.
  • The floors are unchanged: 16 GiB on the four discrete lanes, 8 GiB on the two Strix lanes.
  • The Windows blocks still check memory only. The PR documents this and pins it with a test.
  • At review time, CI on 63cacde0 was still running. Required checks should be green before merge.

What this covered

I read all 6 changed files completely at 63cacde0, against the merge base. The change was read in two parts: the original diff at 492c4e73, then the incremental 492c4e73..63cacde0 (three commits touching .github/scripts/gpu-preflight.sh, docs/testing.md and xtask/src/workflow_contract.rs), the latter by me directly rather than a worker.

What I ran, from a clean git archive of the head:

  • cargo test -p xtask workflow_contract: 60 passed.
  • bash scripts/reclaim-gpu.sh --self-test and the original script experiments, on the earlier head.
  • Mutations on the new head, all caught: dropping the timeout on --assert-clear (2 tests red), swallowing the 124 status (2 red), changing the 124 branch (2 red), and -ge to -gt in the floor comparison (1 red).
  • On the earlier head, also: if true, || true, and the floor changed in each workflow, all caught.

Did not run: shellcheck and shfmt (not installed here); the self-hosted GPU lanes; the Windows and WSL blocks, which are unchanged apart from comments, so I read them only; the history, prior-changes and instruction-adherence passes as separate workers.

Not reconciled against any review threads or earlier discussion. The Jira ticket named in the original request (ROCMAI-446) concerns the coverage gate and does not correspond to this change, which names GitHub issues #446 and #294.

@r0x0r
r0x0r dismissed juhovainio’s stale review October 14, 2026 13:16

Addressed at 63cacde. You were right, and the measurement was worse than the report: with --assert-clear stubbed to stall, the unbounded version at b65b697 exited 0 after 60s — it did not merely ride to the job timeout, it PASSED the gate, asserting a clean host it had never verified. Now timeout 15 bash "${RECLAIM_SCRIPT}" --assert-clear, with 124 handled as a distinct third outcome that fails with its own ::error:: ("could NOT be shown clear"), separate from both pass and survivor-found. Same stub now exits 1 at 15s. Both mutations die: dropping the timeout fails at 60s, 124 -> 125 fails at 15s. Your two non-blocking items are deliberately left (no live-hardware validation; the #294 deltas).

@r0x0r
r0x0r removed the request for review from volen-silo October 14, 2026 13:17

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 63cacde

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Round note: every finding below is new (first round on this PR). This round files a change request on the second and third blocking findings (contract tests that stay green with the guarded code mutated or deleted, and new code paths no test reaches). The first blocking finding (self-hosted GPU lanes not yet run: E2E self-hosted run 38563490556 is pending with zero jobs) is missing evidence, not a defect in the change; it does not gate on its own. The description finding alone would not gate either, because the squash commit is built from commit messages.

Review — needs work

Full review of the whole change.
Closes #446 (adds the process-based option 3 from that issue and keeps the floor, as the issue suggests) and does the preflight part of #294; the rest of #294 (report upload, reclaim, bootstrap dedup) is not attempted, and the PR says "Advances" rather than closes, which is accurate.

Blocking

  • (new) The required self-hosted GPU lanes have not run for this head — E2E self-hosted run 38563490556
    For 63cacde0 this workflow is pending with zero jobs. Only the hosted CI and CodeQL runs completed. The serve path filter matches this PR (scripts/**, .github/workflows/**), so the GPU lanes are expected to run but have not. The PR's main deliverable is a script that gates every native Linux GPU lane, and so far it has only run against stubbed rocm-smi and stubbed reclaim. Four of its six callers are nightly.yml lanes that never run on a PR.
    Confidence 100 · architectural · Fix: get the self-hosted run to actually start (cancel the stalled run and re-run), and dispatch nightly.yml on this branch before merge.

  • (new) Contract tests claim guarantees they cannot fail on — xtask/src/workflow_contract.rs:1265, :1281-1300, :1361-1375, and is_apu_preflight at :1181
    Each of these was mutation-tested, and all tests stayed green:

    • The floor check is step.contains("GPU_PREFLIGHT_MIN_FREE_GIB: 16"). A substring match passes for 160.
    • The floor check and is_apu_preflight both match comment text inside the step. Replacing an APU lane's env: key with # GPU_PREFLIGHT_MIN_FREE_GIB: 8 passes, and that lane then silently runs at the script's 16 GiB default. That is exactly what the assertion message says it prevents ("silently inherit the script's default").
    • every_native_linux_preflight_delegates_in_the_same_shape compares only the run: body. Its rationale names "a lane-local retry, a second floor" as the drift it guards against, yet each of these passes on a single lane: continue-on-error: true, if: false, GPU_PREFLIGHT_CEILING_SECS: 1. continue-on-error: true would make a required gate advisory.
    • In the_shared_preflight_asserts_no_leaked_engine_process_survives, the first assertion code.contains("--assert-clear") can never fail. Comment lines are filtered out, but the non-comment echo "::error::… (reclaim-gpu.sh --assert-clear) did not finish…" at gpu-preflight.sh:246 still contains the string. The test's own comment (":1361") warns of exactly this, and the if bash … --assert-clear; then shape it describes has not existed since 1cde744. With the call deleted, the test fails on a different assertion (the timeout literal), not on the one claiming "no longer RUNS".

    Confidence 100 (each mutation was run) · logic · Fix: parse the step's env: mapping and compare the floor value exactly; pin the non-run: keys of the six steps as equal; make the assert-call check match the call itself (timeout 15 bash "${RECLAIM_SCRIPT}" --assert-clear).

  • (new) New code paths no test reaches — scripts/reclaim-gpu.sh (--assert-clear) arm in main), .github/scripts/gpu-preflight.sh:247-248, :254, :267

    • The xtask harness always stubs reclaim. The self-test (check 14) calls assert_clear directly. So nothing exercises bash reclaim-gpu.sh --assert-clear as the preflight invokes it. With the dispatch arm removed, every test stays green. The script then exits 2, which the preflight reports as "a leaked E2E engine process is still holding it (listed above)" with nothing listed. Exit 127 (script not found) gets the same message.
    • Removing the newly added timeout 15 from either --report-holders call also leaves every test green. The stubs return instantly, so no test can observe these bounds, even though the script's comments argue they are load-bearing.

    Confidence 100 · logic · Fix: have one harness case run the real reclaim-gpu.sh --assert-clear (scoped through RECLAIM_SELFTEST_SCOPE to a decoy); give the "leaked process" message only for exit 1 and a generic message for any other non-zero exit; add a slow --report-holders stub case.

  • (new) The PR description misstates the change — PR body

    • The "Falsified" section describes a mutation of if bash "${RECLAIM_SCRIPT}" --assert-clear; then. That code no longer exists.
    • Two later commits (1cde744, 63cacde) are not mentioned. They introduce a new failure mode: a --assert-clear scan that does not finish within 15 s now fails the lane with its own message.
    • The body also ends with a "Generated with Claude Code" footer. AGENTS.md §11 says "avoid AI-generated boilerplate footers".

    Confidence 85 · mechanical · Fix: update the description to cover the bounded-scan behaviour and the current mutation evidence, and drop the footer.

Non-blocking

  • (new) Gate 2 runs once, and the argument for running it only after the floor does not hold on the discrete lanes — .github/scripts/gpu-preflight.sh:205-260, scripts/reclaim-gpu.sh (reclaim, the SIGKILL loop)

    • The script comment reads: "By here the memory is back, so anything the reclaim rules still match is a process that survived being reclaimed." The PR's own table says the floor only trips near exhaustion (past ~176 GiB held on MI300X). So on those lanes, passing the floor says nothing about whether a process reclaim just SIGKILLed has finished exiting.
    • reclaim() returns right after sending SIGKILL, without waiting for the process to exit.
    • On a single sighting, gate 2 exits 1 with no retry inside the remaining ceiling.
    • Separately, the scan forks tr once for every /proc entry. On this dev box (~900 pids, load ~28) --assert-clear took 16–19 s. A full preflight with a healthy stubbed GPU and no leak then failed with the 124 message, which names a D-state holder as the cause even though no process was stuck.
    • CI exposure is not measured; native hosts that see the full host /proc (the nightly Strix lane) are the likeliest to hit it.

    Confidence 70 · logic · Fix: poll gate 2 until the existing deadline, as gate 1 does; read cmdlines with a builtin (read -r -d '' from /proc/$pid/cmdline) instead of forking tr; word the 124 message as one possible cause, not the cause.

  • (new) WSL comments edited by this change contradict their own step — .github/workflows/e2e-selfhosted.yml:1091-1095, :1231-1233; .github/workflows/nightly.yml:~1171-1176, :1309-1311

    • The new comment says the script is not invoked because "the repo checkout is not at a path this step can assume". The same body runs (cd /root/work/rocm-cli && bash scripts/reclaim-gpu.sh --report-holders). The code is what governs, so --assert-clear could run there the same way. The comment also sets up the stated follow-up on a false premise.
    • The unchanged lines saying "the other preflight blocks are plain run: steps … bash -e {0} … no pipefail" are now stale, because those lanes run set -euo pipefail through the script.
    • A dangling line, # \rocm-smi` itself isn't, is left at e2e-selfhosted.yml:1095`.

    Confidence 85 · mechanical · Fix: correct the reason given (or run --assert-clear from the clone), and update the pipefail sentence.

Decisions for the author

  • Discrete lanes now run the APU superset — tradeoff
    The four discrete bodies were the older single-pool version. They now:

    • try the two-pool query first;
    • require the (B): unit;
    • print a pools line;
    • catch a zero total.

    On a wedged driver, each poll costs about 35 s instead of 20 s, so the 90 s ceiling gets about 3 attempts instead of 5 (measured with stubs). One body for six lanes is the point of #294. The cost is fewer retries on a wedged discrete card, so confirm that is acceptable.

  • Shared script rather than the composite action #294 proposes — tradeoff
    The script keeps job headers untouched and can be tested from a scratch tree. A composite action would also absorb the reclaim, upload and bootstrap duplication #294 measures. The script is a reasonable first step; just confirm that #294 stays open for the rest.

Positive signals

  • --assert-clear reuses select_leaked rather than adding a second matcher, so the gate and the reclaim cannot drift apart. Self-test check 14 drives it in both directions, against a decoy spawned after the reclaim, and asserts the decoy is not killed.
  • Every one of the six removed inline bodies was checked against the shared script: none differed in floor, ceiling, working directory or step environment beyond the intended changes, and the nightly Strix lane's job-level TMPDIR still reaches both reclaim and assert.

Deployment notes

  • The first real run of the process gate will be on hardware. A leaked or slow-exiting engine process left over from an earlier run will now fail a lane that previously passed. Check the first few red preflights before reading them as regressions.

What this covered

Every file in the change, read completely:

  • .github/scripts/gpu-preflight.sh
  • the scripts/reclaim-gpu.sh diff
  • all preflight and reclaim hunks in both workflows
  • the xtask/src/workflow_contract.rs diff
  • docs/testing.md

Plus the surrounding reclaim-gpu.sh functions, the e2e-selfhosted.yml path filter, and issues #446 and #294. Diff prw-base...HEAD = 21be003…63cacde0.

What was run:

  • bash scripts/reclaim-gpu.sh --self-test: 36 ok.
  • cargo test -p xtask --bins -- workflow_contract::tests: 60 passed, all 10 relevant tests named, with pwsh and timeout present.
  • shellcheck 0.11.0 with the repo profile: clean.
  • Stubbed rocm-smi and reclaim scenarios for every preflight branch.
  • Revert and per-branch mutation runs of the contract tests.

The two fan-out workers mutated the same checkout at the same time. One reverted the other's "default floor 16→1" mutant mid-run, so that single mutation result is unreliable; it concerns an unexercised default and is not used in any finding.

Did not run:

  • Self-hosted GPU lanes: no hardware here, and the PR's run has not started.
  • Windows-target compile of xtask.
  • Prior-changes pass: review comments on earlier PRs to these files were not read.

Not reconciled against any discussion here.

r0x0r added 2 commits October 14, 2026 21:12
`timeout` passes the command's own status through, so the preflight sees
127 when `reclaim-gpu.sh` is missing or unreadable and 2 when the two
scripts disagree about a flag — deleting the `--assert-clear` dispatch arm
produces exactly the 2. Both were reported as "a leaked E2E engine process
is still holding it (listed above)", with nothing listed above: every lane
fails, for a reason that is not true, naming a process that does not exist.

Claim a survivor only for exit 1, the one code that means the scan ran and
found somebody, and give every other non-zero exit its own message saying
the host was shown nothing either way.

Also correct the gate-ordering comment. It claimed "by here the memory is
back, so anything the reclaim rules still match is a process that survived
being reclaimed" — but the floor is 8/16 GiB against pools of ~62 and ~192
GiB, so on a healthy host it is met on the first poll with no wait at all.
Clearing it therefore says nothing about whether a SIGKILLed process has
finished exiting; what bounds that is the reclaim's own grace period. The
ordering is still the right way round, it is simply not the proof of a
settled card the comment claimed, and a process still dying when this runs
is a false positive this gate can produce.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Each of these was mutation-tested and each stayed green on the mutation its
own message says it prevents.

The floor check was `step.contains("GPU_PREFLIGHT_MIN_FREE_GIB: 16")`. That
is a substring of `: 160`, and it searched the whole step, so commenting the
key out satisfied it too — on an APU lane that leaves the 8 GiB floor off
and the script's discrete 16 GiB default on, which is verbatim what the
assertion message says it stops. Parse the step's `env:` mapping and compare
the value exactly; `is_apu_preflight` reads the same parsed value, and its
two body spellings now match with comments stripped.

`every_native_linux_preflight_delegates_in_the_same_shape` compared only the
`run:` body, while its rationale named "a lane-local retry, a second floor"
as the drift it guards. `continue-on-error: true` on one lane turns a
required gate advisory, `if: false` removes it, and a lane-local
`GPU_PREFLIGHT_CEILING_SECS` is a second gate — none touch the body, all
three were green. Pin each step's keys to `name`/`env`/`run` and its step
environment to the floor alone, then compare the whole step around the body
across the six, so a lane cannot drift in either direction.

In `the_shared_preflight_asserts_no_leaked_engine_process_survives`,
`code.contains("--assert-clear")` could not fail. Filtering comment lines
fixed that once; the timeout fix then added a non-comment `echo "::error::…
(reclaim-gpu.sh --assert-clear) did not finish…"` and defeated the filter
again. Match the invocation itself — a non-comment line calling the script
with the flag — and require every such line to be the bounded form, so
neither deleting the call nor guarding it can stay quiet.

Add the three cases nothing reached. The real `reclaim-gpu.sh` is now
planted instead of the stub in one case, driven in both directions against a
decoy scoped by `RECLAIM_SELFTEST_SCOPE`: with the dispatch arm deleted the
harness always stubbed reclaim and the self-test called `assert_clear` as a
function, so nothing crossed the process boundary the lane crosses. Exit 2
and 127 are driven for the message they now get, and both
`--report-holders` call sites against a stub that sleeps past their bound.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — all four blocking findings reproduced exactly as described. Fixed in 56cff06c and da245d60.

Every mutation was re-run before and after, on the whole workflow_contract module:

mutation before after, caught by
a discrete lane's floor 16 → 160 green …delegates_to_the_shared_script, "without naming its own floor (found Some("160"))"
an APU lane's floor key commented out green same, plus …delegates_in_the_same_shape, …measures_the_gtt_pool, apu_preflight_twins_do_not_drift
continue-on-error: true on one lane green …delegates_in_the_same_shape, "is no longer just a named, floor-carrying delegation"
if: false on one lane green same
a lane-local GPU_PREFLIGHT_CEILING_SECS: 1 green same, "sets step environment beyond its floor"
--assert-clear call line deleted red, on the timeout literal …asserts_no_leaked_engine_process_survives, on "no longer RUNS" — plus the four runtime cases
that call wrapped in if false; then … fi red, on the timeout literal same test, on "does not invoke … as a plain, timeout 15-bounded command"
--assert-clear) arm deleted from reclaim-gpu.sh's main green — suite and the 36-check self-test preflight_runs_the_real_reclaim_assert_clear
timeout 15 removed from the gate-2 holder report green preflight_bounds_the_holder_report
timeout 15 removed from the ceiling holder report green same test, other case
every non-124 failure reported as a survivor n/a (new branch) preflight_distinguishes_a_scan_that_never_ran_from_a_survivor

On the vacuous assertion, confirmed directly rather than inferred: with the call line deleted and comment lines filtered, the one surviving non-comment line containing the flag is

echo "::error::GPU preflight failed: the leaked-process scan (reclaim-gpu.sh --assert-clear) did not finish within 15s …"

so code.contains("--assert-clear") was true. You were right that this is the same defect twice: 1872c76e fixed it by filtering comments, 1cde744 reintroduced a non-comment occurrence. It now matches the invocation — a non-comment line containing bash "${RECLAIM_SCRIPT}" --assert-clear — and requires every such line to begin timeout 15 bash.

What changed, briefly:

  1. step_env() parses the step's env: mapping; the floor is compared by value (Some("8" | "16")), and is_apu_preflight reads the same parsed value with its two body spellings matched over comment-stripped body code.
  2. …delegates_in_the_same_shape now pins each step's keys to exactly name/env/run, its step environment to the floor alone, and compares the whole step around the body across the six — so single-lane and all-six drift both fail.
  3. gpu-preflight.sh claims a survivor only for exit 1; 124 keeps its own message and everything else gets a third one naming the code and saying the host was shown nothing either way.
  4. Three new runtime cases: the real reclaim-gpu.sh driven through the preflight in both directions against a decoy scoped by RECLAIM_SELFTEST_SCOPE; exit 2 and 127; and both --report-holders sites against a stub that sleeps past their bound.
  5. Non-blocking ordering comment corrected — it claimed "by here the memory is back", which, as you note, the floor does not establish on a lane where it is met on the first poll. It now says the ordering is right but is not proof of a settled card, and that a still-dying process is a false positive this gate can produce.
  6. PR description rewritten: the stale if bash … --assert-clear; then mutation is gone, the bounded-scan failure mode from 1cde744/63cacde is described, and the footer is removed (AGENTS.md §11).

Devbox: cargo test -p xtask 259 passed / 0 failed (was 256); cargo clippy --workspace --all-targets -D warnings clean; cargo fmt --check clean; shellcheck 0.10.0 with the repo profile and bash -n clean on both scripts.

Two observations, neither a change here:

  • The self-hosted lanes still have not run. E2E self-hosted 38563490556 is still pending with zero jobs, 29 h after creation; the repo's self-hosted queue is 78 runs deep. That reads as the backlog, not this branch. Agreed it is an evidence gap rather than a defect, and I have not treated it as one.
  • Your --assert-clear timing: on this devbox (112 cores, 2424 pids, load ~12) three consecutive runs took 4 s each, not 16–19 s. The fork-per-/proc-entry cost is real and your reasoning stands, but the margin against the 15 s bound looks wider than your measurement suggested. The new real-script case detects the 124 message and skips with a stated reason rather than reporting a loaded host as a broken gate. Separately: scripts/reclaim-gpu.sh --self-test fails one unrelated check here (report_holders did not list the manual serve) identically at 63cacde0 — report_holders pipes ps -eo through head -40, which truncates on a box this busy. Pre-existing and out of scope for this PR.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

# Conflicts:
#	.github/workflows/nightly.yml
#	xtask/src/workflow_contract.rs
@r0x0r
r0x0r added this pull request to the merge queue Oct 15, 2026
Merged via the queue into main with commit e3c09c5 Oct 15, 2026
24 checks passed
@r0x0r
r0x0r deleted the ci/rocmai-446-gpu-preflight-process-check branch October 15, 2026 07:01
@nowycondro nowycondro mentioned this pull request Oct 27, 2026
3 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci(e2e): GPU preflight free-memory floor is a fixed 8/16 GiB regardless of pool size

5 participants