Skip to content

ci: run the GPU preflight before each queue-lane channel - #999

Merged
tomastola merged 4 commits into
mainfrom
ci/queue-lane-preflight-per-channel
Nov 11, 2026
Merged

tomastola merged 4 commits into
mainfrom
ci/queue-lane-preflight-per-channel

Conversation

@tomastola

@tomastola tomastola commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator
  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. — not a bug fix in the product
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. — no subcommand; CI only
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. — no product messages change

Why

Follow-up to #950, from its review. The queue lane runs both channels one after the other on the same GPU, but the GPU preflight ran once, before the build. The second channel therefore started without the preflight and stray-server reclaim that each channel's job in nightly.yml gets, after the first channel's suite had run.

What changes

  • scripts/run-e2e-lane.sh runs the preflight before each channel after the first; the first channel keeps the one before the build, so none runs twice. A failing preflight fails that channel and the next still runs, as for any other failure in a channel.
  • The lane test now requires one preflight per channel, in order, and fails without the new call.
  • docs/nightly-on-the-queue.md says an incomplete lane is possible while the two-channel time is unmeasured, points at the run's wait input and its limit, and says each channel starts with the GPU preflight.
  • The contract test that keeps other workflows off the queue runner matches the label and the two workflow names as whole words, so queue-runner is not read as queue-run; the workflow's display name is matched as before.

Verification

  • cargo test -p xtask: 760 pass; cargo clippy -p xtask --all-targets -- -D warnings clean; the lane script passes the repository's shellcheck settings.
  • Removing the new preflight call makes the lane test fail.
  • Not verified: a lane on the queue with this change; the first queue run started from main runs the script as merged in ci: run the nightly GPU E2E lanes on a shared GPU queue #950.

Scope

CI scripts, one doc paragraph and one contract test only. No existing lane or required check changes.

…ueue names as words

- run-e2e-lane.sh runs the GPU preflight again before each channel after the
  first, as each channel's job in nightly.yml does, so a model server left by
  the previous suite is reclaimed; the lane test now requires one preflight
  per channel.
- The doc gives the default wait (14400 s).
- The contract test matches the queue label and workflow names as whole
  words, so `queue-runner` is not read as `queue-run`.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@tomastola

Copy link
Copy Markdown
Collaborator Author

One failing check, unrelated to this change: E2E consolidated report (self-hosted) fails with lane e2e-gpu-w7900 ran (needs result: cancelled) but uploaded no e2e-gpu-w7900-report. The E2E tests (W7900, current) job never got a runner: it sat queued for 24 h (no runner assigned) and was cancelled by the time limit, so it produced no report. The w7900-0 runner is online but busy, and the same lane is still queued, unstarted, on the latest main run (run 42889292440; its MI300X lane is queued too). The other lanes (Strix Halo Ubuntu release and nightly, Windows, WSL2) passed on this head. Re-running the job once the W7900 runner is free should clear it; I have not re-run it.

@tomastola
tomastola marked this pull request as ready for review November 10, 2026 22:14
@tomastola
tomastola requested a review from a team as a code owner November 10, 2026 22:14

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 2c13cd6f

Review — needs work

Full review of the whole change (3 files: scripts/run-e2e-lane.sh, docs/nightly-on-the-queue.md, xtask/src/workflow_contract.rs).
Follow-up to #950 per its own review; no ticket named beyond that PR reference. The description's claimed scope (per-channel preflight, doc default-wait sentence, whole-word queue-reference matching) matches what the diff actually does.

Blocking

  • A failed per-channel preflight leaves the previous channel's stale results reported under the new channel's name — scripts/run-e2e-lane.sh:150-158, xtask/src/workflow_contract.rs (lane test harness around run_lane/LaneCase, ~9308-9560)
    In run_channel(), the newly-added preflight call (if [[ "${preflight_done}" == 1 ]]; then bash .github/scripts/gpu-preflight.sh || return $?; fi) sits before the pre-existing rm -rf tests/e2e-cucumber/results || return $? a few lines down. If the second channel's preflight fails — exactly the case this PR exists to catch, a model server left on the GPU by the first channel's suite — run_channel returns early via return $? without ever reaching the rm -rf. Back in the outer loop, save_results "${channel}" still runs unconditionally and copies whatever is sitting in tests/e2e-cucumber/results (the previous channel's untouched output) into ${out}/<new-channel>/results. The new channel is then reported as having produced results it never produced — the previous channel's junit.xml, mislabeled. This directly violates the script's own stated invariant ("a channel must not report the last one's files") and the published-results contract in docs/nightly-on-the-queue.md ("/out/<channel>/results" holds that channel's own suite). Confirmed by running the actual script with stubbed tools: making channel 2's preflight stub exit non-zero reproduces out/nightly/results/junit.xml containing release's runtimes string, with lane exit code 7. An independent blind pass over the same files reached the identical conclusion unprompted. Notably, the base commit's own history (#950) added "keep a failed results cleanup loud" specifically to stop a prior version of this exact class of bug — this change reopens it through a different path.
    No test exercises this path: LaneCase's stubbed .github/scripts/gpu-preflight.sh always succeeds, so the new preflight_done branch is only ever tested on its happy path.
    Confidence 100 · logic · Fix: move the rm -rf tests/e2e-cucumber/results (and clearing any prior E2E_SHARED_RUNTIMES_DIR/prewarm state) ahead of the per-channel preflight check, or skip save_results for a channel whose run_channel returned before ever reaching its own prewarm/suite step. Add a LaneCase option to fail the stubbed preflight on a chosen call and assert the failed channel's output directory is absent or empty, not a copy of the previous channel's.

Non-blocking

None

Decisions for the author

None

Positive signals

  • queue_runner_references's new whole-word matching (xtask/src/workflow_contract.rs, word boundary on -/_/alphanumerics) is well targeted at the false positives it names (queue-runner, queue-reporter) without loosening the real syntactic forms (uses:, needs:, label arrays) that actually create a dependency on the queue workflows; the added test cases (single- and multi-line needs:, the two explicit non-matches) cover the boundary cleanly.
  • The doc's new default-wait sentence is appropriately hedged ("unless a run sets another") rather than overstating a value that actually lives outside this repository.

Deployment notes

None beyond the bug above — this is CI-only tooling, no release/runtime impact.

What this covered

Read in full: scripts/run-e2e-lane.sh, docs/nightly-on-the-queue.md, and the relevant sections of xtask/src/workflow_contract.rs (the full queue_runner_references function and its test, and the full lane-script test harness: run_lane, LaneCase, and every a_lane_*/a_failing_channel_* test), against base 07c93f323513a0d0b85af19303f67ddba54c90d9 → head 2c13cd6f13428f86064a84be4cc6ba5cb3a897de. The blocking finding was verified by actually running the script with stubbed cargo/apt-get/gpu-preflight.sh and forcing the second preflight call to fail, and independently corroborated by a second, blind pass over the same files. CI: all required checks pass on this head; the one failing check (E2E consolidated report (self-hosted)) traces to the E2E tests (W7900, current) job sitting queued for 24h with no available self-hosted runner and being cancelled on timeout — unrelated to any file this PR touches (run-e2e-lane.sh is not referenced anywhere in e2e-selfhosted.yml), and that workflow's own header states its lanes are deliberately excluded from branch protection's required-check list.
Did not run: a live queue lane (not reachable from here; the author's own PR text already flags this as unverified). Prior-changes pass found no open review comments on earlier changes to these files (PR has no prior reviews). Not reconciled against PR discussion beyond the description itself.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

A failed preflight returned before the results cleanup, so save_results copied
the previous channel's output under the failed channel's name. The cleanup now
runs first; a lane test fails the second preflight and requires that channel to
report nothing.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola

Copy link
Copy Markdown
Collaborator Author

Addressed the review's blocking finding in 6356b11: run_channel now clears tests/e2e-cucumber/results before the per-channel preflight, so a failed preflight no longer leaves the previous channel's results to be saved under the new channel's name. New lane test a_channel_whose_preflight_fails_does_not_report_the_last_ones_results fails the second preflight and asserts the nightly channel reports nothing; it fails on the old script and passes on the fix. The W7900 / consolidated-report failure is unchanged (runner never assigned).

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 6356b11

Review — no blocking findings

Full review of the whole change (3 files: scripts/run-e2e-lane.sh, docs/nightly-on-the-queue.md, xtask/src/workflow_contract.rs).
Follow-up to #950 per its own review; no ticket named beyond that PR reference. The description's claimed scope (per-channel preflight, doc default-wait sentence, whole-word queue-reference matching) matches what the diff actually does.

Blocking

None

Non-blocking

  • A function doc comment over-claims a uniform whitespace-stripped matching strategy that no longer holds for two of the three patterns it covers — xtask/src/workflow_contract.rs:9012-9016 (doc comment) vs. the implementation at :9029-9039
    The doc comment above queue_runner_references still says matching is done "Compared with case, quotes and every whitespace character (line breaks included) removed." That is only true of the workflow-name branch (still squeezed). The three literal words (queue-report, nightly-queue, queue-run) are now matched via names_word against lowered, which strips only case and quotes, not whitespace, and applies word-boundary matching instead. The newer, correct explanation exists as an inline comment a few lines below, but the top-of-function doc comment was not updated to stop over-claiming for the three-word path. Confirmed by direct comparison against the base version, where this comment was accurate (the old code squeezed everything uniformly).
    Confidence 85 · mechanical · Fix: rewrite the doc comment to describe both comparison strategies (word-boundary on unsqueezed, case/quote-stripped text for the three literal words; whitespace/quote-stripped substring match for the workflow name).

  • The dynamic workflow-name match keeps the old unguarded substring check the word-boundary fix was introduced to avoid — xtask/src/workflow_contract.rs:9046-9049
    This diff adds word-boundary protection so queue-run no longer matches inside queue-runner/queue-reporter (confirmed by the new test cases). The workflow-name branch (squeezed.contains(name.as_str())) is unchanged and still does a bare substring check; simulated against a hypothetical workflow whose squeezed display name is a superstring of a monitored one (e.g. "Nightly E2E (queue) debug" vs. "Nightly E2E (queue)"), it would be falsely flagged. This is pre-existing (identical substring logic existed before this PR for the name branch) and no current workflow in .github/workflows/ triggers it, so it is not a regression this change causes — flagged as pattern-relevant since this PR fixes the identical class of bug for the other three patterns in the same function.
    Confidence 75 · logic · Fix (optional follow-up): apply an equivalent boundary check to the name match, or treat workflow_run.workflows/needs/label-array delimiters explicitly.

Decisions for the author

  • The new default-wait figure ("14400 s, 4 h") has no in-repo test backing it — tradeoff
    Unlike the run's upper limit (15000 s / 4 h 10 min), which a contract test cross-checks against queue-run.yml, the per-lane default wait is owned by the external queue system and isn't present anywhere in this repository (checked queue-run.yml and nightly-queue.yml directly). That makes it untestable from here by construction, not an oversight — the sentence is appropriately hedged ("unless a run sets another"). Worth the author's own confirmation that 14400 s is in fact the queue side's current default, since nothing here can check it.

Positive signals

  • The fix correctly reorders rm -rf tests/e2e-cucumber/results ahead of the per-channel preflight check in run_channel(), closing the exact gap a prior review round found (a failed second-channel preflight no longer leaves the first channel's stale results copied under the new channel's name), and ships a regression test (a_channel_whose_preflight_fails_does_not_report_the_last_ones_results) that fails on the old ordering and passes on the fix — traced by hand across every control-flow path (first/second channel, preflight success/failure) and reproduced standalone.
  • names_word's boundary check correctly handles matches flush against the start or end of the text (an easy off-by-one to miss, and it wasn't missed here).
  • Comments in run-e2e-lane.sh correctly call out subtle bash semantics worth preserving (set -e suppression inside the run_channel "${channel}" || rc=$? caller, and why preflight_done is deliberately not local).

Deployment notes

None — CI-only tooling (a self-hosted lane script and its contract tests), no release or runtime impact.

What this covered

Full diff read against base 07c93f323513a0d0b85af19303f67ddba54c90d9 → head 6356b115965bedce46674653b1890a8e5495787c (two commits: 2c13cd6f, 6356b11): scripts/run-e2e-lane.sh in full, docs/nightly-on-the-queue.md in full, and the relevant sections of xtask/src/workflow_contract.rs (the complete queue_runner_references function and its test, and the complete lane-script test harness — LaneCase, run_lane, and every a_lane_*/a_failing_channel_*/a_channel_whose_* test). Reviewed as two independent fan-out passes grouped by coupling (lane script + lane test harness; queue-name contract test + doc paragraph), consolidated and adversarially re-verified here, including reading .github/scripts/gpu-preflight.sh directly to confirm it has no dependency on per-channel environment variables, and reading the repo's actual .github/workflows/*.yml display names to check the superstring-match concern above against real data.

Did not run: cargo test -p xtask / cargo clippy / shellcheck (no Rust or shellcheck toolchain available in this environment) — the PR's own verification section claims these pass; not independently re-executed here, though the specific control-flow claim about preflight-failure/results-clearing was reproduced with a standalone shell script. CI: all required checks pass on this head. One self-hosted lane (E2E tests (Strix Halo, WSL2, current)) failed with a transient WSL/RPC provisioning error ("the RPC server is unavailable") unrelated to any file this PR touches; two self-hosted lanes (MI300X, W7900) remain queued. e2e-selfhosted.yml's own header states none of its lanes are in branch protection's required-status-check list, so none of this blocks merge.

A prior review round (at commit 2c13cd6f, since superseded) is recorded on this PR; its one blocking finding concerned exactly the ordering bug this head's second commit fixes, used here only as the method's "prior changes" source-pass, not as an anchor for this round's findings — this round's independent fan-out pass over the fixed code reached its own conclusions by tracing and reproducing the control flow directly. Not otherwise reconciled against PR discussion beyond the description itself.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing here looks wrong in the change itself: run_channel runs in the same shell as the loop, so preflight_done carries across channels, the word-boundary matching in workflow_contract.rs is safe, and a failed preflight still lets the next channel run. The diff is clean on the leak scan, and being CI-only it needs no scenario.

I'm requesting changes for two small things to tighten before merge, both left as inline comments:

  • The new test covers the second (nightly) channel's preflight failing, but not the first, pre-build one, which now takes a different code path.
  • The docs sentence adds a "14400 s, 4 h" default wait that I couldn't find defined in this diff or in scripts/run-e2e-lane.sh; please name where it comes from. The paragraph is also rewrapped unevenly.

On CI: E2E tests (Strix Halo, WSL2, current) is failing after about 40 s and I couldn't see its log, so I can't say whether it is lane infrastructure or this change. The MI300X and W7900 lanes are still pending.

Comment thread docs/nightly-on-the-queue.md Resolved
Comment thread xtask/src/workflow_contract.rs Resolved
…ment

- A lane test fails the first (pre-build) preflight and requires that nothing
  builds, runs or reports.
- The queue doc no longer states a 14400 s default wait: it is not defined in
  this repository; the run's wait input and its 15000 s limit are named
  instead, and the paragraph is rewrapped evenly.
- The doc comment on queue_runner_references says which patterns are matched
  as whole words and which with whitespace removed.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola requested a review from juhovainio November 11, 2026 08:45

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 320aa393

Review — needs work

Full review of the whole change (3 files: scripts/run-e2e-lane.sh, docs/nightly-on-the-queue.md, xtask/src/workflow_contract.rs).
Follow-up to #950, from its review; no ticket named. The description's claimed scope (per-channel preflight, the doc's wait paragraph, whole-word queue-reference matching) matches what the diff actually does, and the bundled whole-word-matching fix is disclosed in "What changes" rather than hidden.

Blocking

  • The windows-build-and-test gate (and the underlying Windows build and test (debug) job) is failing at this head — CI, not a file in this diff
    The failing test is therock::python::config_props::managed_python_recording_never_replaces_an_unreadable_config_with_defaults, in code this PR does not touch (the diff only changes scripts/run-e2e-lane.sh, docs/nightly-on-the-queue.md, xtask/src/workflow_contract.rs). The same job succeeded on the current tip of main, which points at an environmental flake rather than a regression this change introduces — but that is a thing to confirm by re-running, not something I can establish from here, and per AGENTS.md §12 ("Drive To Green" / "handle infrastructure flakes by rerun or maintainer escalation with evidence") a red check stays "not ready" until it's actually resolved one way or the other.
    Confidence 100 · logic · Fix: re-run the Windows debug job; if it goes green, this was a flake and needs no code change here. If it stays red, it is unrelated to this PR's content and should be escalated/fixed separately rather than blocking this merge on an unrelated failure.

Non-blocking

  • queue_runner_references leaves the dynamic queue_workflow_name match as an unguarded, whole-document substring check — the same bug class the PR just fixed for the other three patterns — xtask/src/workflow_contract.rs:9047-9050
    The three literal words (queue-report, nightly-queue, queue-run) now go through names_word's word-boundary check, so queue-runner/queue-reporter no longer false-match. The workflow-name branch is untouched: squeeze(queue_workflow_name) is still checked with a plain squeezed.contains(name.as_str()), with whitespace stripped across the whole document and no boundary protection. No workflow in this repo currently collides (checked the real .github/workflows/*.yml name: fields), so nothing fails today, and the final commit's doc-comment rewrite shows this asymmetry was a conscious, documented choice rather than an oversight. It is still the same fragility this PR set out to close, left open for the fourth pattern: a future rename of either queue workflow to something shorter/more generic, or an unrelated adjacent token sequence elsewhere that squeezes into the same substring once a comment line is dropped, would reopen it.
    Confidence 70 · logic · Fix (optional follow-up): give the name match the same names_word boundary check, or scope it more tightly than a whole-document substring search.

  • The function's doc comment repeats a "comments do not count" claim that without_comments does not actually guarantee — xtask/src/workflow_contract.rs:9014 (doc comment) vs. without_comments at :751-756
    without_comments only drops a line whose first non-whitespace character is # (a whole-line comment); it does not strip a trailing # ... after real content on the same line. So a workflow line such as runs-on: ubuntu-latest # previously ran on queue-report, moved off still contains the literal text queue-report after without_comments runs, and queue_runner_references would report it as a reference even though it is prose in a trailing comment. This is pre-existing — without_comments itself is untouched by this diff, and the old pre-fix code called it the same way for all four patterns — but this PR's own final commit rewrites this exact doc comment and keeps the inaccurate "Comments do not count" wording verbatim, so the claim this diff ships is still wrong for a trailing-comment case the existing tests don't exercise (the suite's own negative comment case is a whole-line comment, which is handled correctly).
    Confidence 85 · mechanical · Fix: narrow the doc comment to "whole-line comments do not count," or switch the function to the file's line-scoped strip_comment helper if the stronger guarantee is actually wanted.

Decisions for the author

  • A single non-local preflight_done shell variable, mutated inside run_channel() and read back by the next call from the outer loop, versus checking "is this the first requested channel" explicitly — tradeoff
    The current approach is correct as written — run_channel is always invoked as a plain command in the main shell's for loop, never in a subshell/pipeline/$(...)/background job, so the mutation reliably persists across calls, and all four control-flow cases (first-channel preflight failing under set -e, a channel's own suite failing, a later channel's preflight failing, a later channel's own suite failing) were traced and are handled correctly, including that rm -rf tests/e2e-cucumber/results now runs before the per-channel preflight check so a failed second-channel preflight can no longer leave the first channel's results reported under the new channel's name. An explicit "am I the first channel in E2E_CHANNELS" check would be less dependent on call-order and non-local mutation surviving future refactors of the loop, at the cost of a bit more plumbing. Worth confirming this was a deliberate simplicity-over-robustness choice rather than an oversight.

Positive signals

  • The preflight/results-ordering fix is correct across all four control-flow cases (first-channel preflight failure, a channel's own suite failure, a later channel's preflight failure, a later channel's own suite failure) — traced by hand against run_channel() and the outer loop.
  • The new regression test (a_channel_whose_preflight_fails_does_not_report_the_last_ones_results) genuinely catches the bug it targets: reverting the fix (moving rm -rf back below the preflight check) makes it fail, confirmed by tracing the reverted control flow against its assertions.
  • The preflight_fails_on counter-file stub in the test harness is off-by-one-safe: the counter starts at 0 and increments before comparison, so the default "never fails" (0) can't accidentally collide with a real 1-based call count.
  • names_word's own word-boundary logic is sound: start-of-text, end-of-text, whole-text matches, and non-ASCII content elsewhere in the workflow text are all handled correctly without a UTF-8 char-boundary panic, since all three literal needles are pure ASCII.
  • The doc's new backward reference ("see the run's inputs above") actually resolves to real, accurate content earlier in the same document (the wait's 15000 s limit in the "What a run executes" list) — not a dangling reference.
  • .github/scripts/gpu-preflight.sh (unchanged, but relied on more heavily by this diff) genuinely performs both a bounded GPU-memory wait and a reclaim-gpu.sh --assert-clear call, so running it again before the second channel actually achieves the stray-server reclaim the PR's rationale claims, not just a superficial check.

Deployment notes

None — CI-only tooling (a self-hosted lane script, a doc paragraph, and contract tests), no release or runtime impact.

What this covered

Full diff read against base 07c93f32 → head 320aa393 (3 commits: 2c13cd6f, 6356b115, 320aa393): scripts/run-e2e-lane.sh in full (both versions), docs/nightly-on-the-queue.md in full (both versions), and the complete xtask/src/workflow_contract.rs (~9800 lines, both versions) — the lane-script test harness (LaneCase, run_lane, every a_lane_*/a_failing_channel_*/a_channel_whose_* test) and the queue_runner_references function with its test table, each read in full by an independent fan-out worker grouped by coupling (lane script + its test harness + doc; the queue-name contract test), consolidated and re-verified here directly against the source. Agent-instruction-adherence ran against this repo's root AGENTS.md (§3 regression-test requirements, §5 doc-citation conventions, §6 module-size test-gated exemption, §11 PR-scope, §12 CI-ownership) — no violation found beyond the CI note above. History/prior-changes pass: all three files were last touched in #950 (this PR's stated predecessor); #950's own review threads raised exactly the two things this PR fixes ("the second channel… gets no preflight or stray-server reclaim" and "the contract test's substring match on queue-run is broad"), and this diff's changes do address both. No separate ticket is named in the title or description beyond the #950 reference, so no tracker lookup applies. CI: read via get_check_runs at this exact head; one gate failing as described above, all other checks (clippy, the Rust/xtask test jobs, CodeQL, security scans, license/lint checks, Linux E2E, Windows release build) green.

Did not run: cargo test -p xtask / cargo clippy / shellcheck locally (no Rust/shellcheck toolchain in this environment) — the PR's own verification section claims these pass; not independently re-executed, though the specific control-flow claims above were traced by hand against the actual source rather than assumed. Did not execute a real queue lane (not reachable from here; the author's own PR text already flags this as unverified). Not reconciled against this PR's existing review discussion: a prior review's login/commit metadata was checked only to confirm no duplicate review already existed at this exact head, never its content, and this report's findings were reached independently of it.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change itself looks right to me: the preflight now runs before each later channel, a failing preflight fails that channel without stopping the next one, and the results directory is cleared first so a channel can't report the previous one's files. The tests assert that state rather than just the wording.

A few things to fix before merge, all small:

  1. The PR description says docs/nightly-on-the-queue.md "gives the default wait, 14400 s", but neither the diff nor the doc at this head contains 14400. The doc only gives the 15000 s limit, and the diff adds just "(see the run's inputs above)". Either add the default to the doc or correct the description.
  2. The comment moved into scripts/run-e2e-lane.sh is a single very long line (inline comment below).
  3. Optional: the doc doesn't say that a preflight now runs before each channel, which is visible in the lane logs. One sentence would cover it.

On CI: the Windows build and test (debug) job fails in therock::python::config_props::managed_python_recording_never_replaces_an_unreadable_config_with_defaults, in a file this PR doesn't touch, so it looks unrelated, but please confirm it also fails on main or rerun it. The E2E checks were still pending when I looked.

Comment thread scripts/run-e2e-lane.sh Resolved
- Wrap the run_channel comment like its neighbours.
- The queue doc says each channel starts with the GPU preflight.
- The doc comment on queue_runner_references says only whole-line comments
  are ignored, as without_comments does.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola

Copy link
Copy Markdown
Collaborator Author

Re-review of 320aa39, handled in 205d9c9:

  • 14400 s in the description: correct, the doc never had it (removed in 320aa39). The description now matches the doc.
  • Wrapped comment in run-e2e-lane.sh: fixed (thread resolved).
  • Doc sentence on the per-channel preflight: added.
  • Doc comment on queue_runner_references: now says only whole-line comments are ignored, as without_comments does.
  • Windows build and test (debug): therock::python::config_props::managed_python_recording_never_replaces_an_unreadable_config_with_defaults failed with os error 3 creating a directory under a shared test temp root, a race inside the test, in code this PR does not touch (it changes only scripts/run-e2e-lane.sh, the doc and xtask). The same job passed at 6356b11 and its test code is unchanged between the two heads; it did not fail in the other red Windows run I checked (ROCMAI-796, 9d132b6). I pushed a new head, so the whole suite re-runs on 205d9c9.
  • Not changed: the workflow-name match in queue_runner_references stays a squeezed substring match (documented; no workflow collides today), and preflight_done stays a shell variable, which is safe as run_channel always runs in the main shell.

@tomastola
tomastola requested a review from juhovainio November 11, 2026 09:29

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. No blocking findings; the queue-lane preflight runs once per channel in order, and the tests cover the second and the first preflight failing. CI is green (the two skipped jobs are the skillscope and Sphinx checks).

Non-blocking:

  • The contract test now matches queue-report, nightly-queue and queue-run as whole words. A positive case for the label inside a flow list or an expression would pin the narrowing from both sides.
  • preflight_done only persists because run_channel is not run in a subshell; a one-line comment would protect that.
  • The docs say the preflight reclaims a stray model server, but the test stub only echoes "preflight ran", so the reclaim itself isn't asserted here.
  • The doc comment on queue_runner_references wraps oddly (cosmetic).

Comment thread xtask/src/workflow_contract.rs Resolved
Comment thread scripts/run-e2e-lane.sh Resolved

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 205d9c97

Review — no blocking findings

Full review of the whole change (3 files: scripts/run-e2e-lane.sh, docs/nightly-on-the-queue.md, xtask/src/workflow_contract.rs). Follow-up to #950, from its review; no separate ticket named. The description's claimed scope (per-channel preflight, the doc's wait/preflight paragraph, whole-word queue-reference matching) matches what the diff does.

Blocking

None

Non-blocking

  • queue_runner_references's dynamic workflow-name match is still an unguarded whole-document substring check, the same bug class this diff just fixed for the other three patterns — xtask/src/workflow_contract.rs:9048-9051
    The three literal words (queue-report, nightly-queue, queue-run) now go through names_word's word-boundary check, so queue-runner/queue-reporter no longer false-match (confirmed by the new test cases). The fourth pattern, the queue workflow's own display name, is unchanged: squeezed.contains(name.as_str()) is a plain substring check over the whole whitespace/quote-stripped document, with no boundary protection. Checked the real workflow names in .github/workflows/*.yml ("E2E on the queue", "Nightly E2E (queue)") — neither is a plausible substring of unrelated text today, so nothing fails currently, and the updated doc comment is honest about this asymmetry rather than silent about it. It's still the same fragility pattern this PR set out to close, left open for the one pattern that can't use names_word as written (the name can contain spaces and parentheses, which the word-boundary helper doesn't handle).
    Confidence 70 · logic · Fix (optional follow-up): scope the name match to known YAML delimiters (uses:, workflows: list entries) rather than a whole-document substring search, or accept this as a documented, lower-priority gap.

Decisions for the author

  • Both the PR description and the doc paragraph describe "a failing preflight" right next to language that, read in isolation, could be taken to cover the very first preflight call (the one before the build, unchanged since #950) — tradeoff/wording
    Ran the script directly with a stubbed preflight set to fail on its first call: the lane exits before cargo build ever runs, no channel is attempted, and /out is never created — matching the new a_lane_whose_first_preflight_fails_builds_and_runs_nothing test, and this is pre-existing behaviour from #950 (the top-level bash .github/scripts/gpu-preflight.sh call has no || guard, so set -e kills the whole script). That's different from a later, per-channel preflight failing, where only that channel fails and the next one still runs — which this PR's "A failing preflight fails that channel and the next still runs" line and the doc's adjacent "Both always run" sentence read most naturally as, if taken as a blanket statement. The code and its test are correct and clearly intentional (the PR's own "Scope" section says no existing lane behaviour changes), so this is a wording-precision point rather than a defect: consider a one-clause carve-out in either place ("...for a preflight after the first; the first failing still aborts the lane before any channel runs, as before") so a reader doesn't have to run the script to find the distinction, the way this review did.

Positive signals

  • The core fix — moving rm -rf tests/e2e-cucumber/results ahead of the per-channel preflight check in run_channel() — was verified by actually running the script (stubbed tools) with the second channel's preflight forced to fail: out/nightly is never created and out/release/results/junit.xml is untouched, so the failed channel reports nothing rather than a copy of the previous channel's results.
  • names_word's boundary check is sound: it only needs the character immediately before/after a match to be a non-word character (or absent), handling start-of-text and end-of-text matches correctly, and the match positions it slices on are guaranteed valid UTF-8 boundaries since they come straight from match_indices on an ASCII needle.
  • The doc comment on queue_runner_references no longer overclaims: it now says "whole-line comments do not count (a trailing # ... after content does)," which matches without_comments's actual behaviour (filters only lines whose first non-whitespace character is #) — a real fix from an earlier round rather than cosmetic.
  • Good new test coverage for the whole-word matching: both the two added positive cases (needs: [queue-run], a multi-line needs: list) and the two explicit non-matches (queue-runner.yml, queue-reporter) pin the boundary from both sides.
  • The comment moved into run_channel() is wrapped consistently with its neighbours, and the PR description's earlier "14400 s" claim (which didn't match the diff) has been removed and replaced with an accurate cross-reference to the run's wait/limit section above it.

Deployment notes

None — CI-only tooling (a self-hosted lane script, a doc paragraph, and contract tests), no release or runtime impact.

What this covered

Reviewed as a single pass (the change is small enough — 3 files, 121+/19− — that Step 3's own sizing guidance calls for one careful pass rather than a fan-out): scripts/run-e2e-lane.sh in full (both base and head), docs/nightly-on-the-queue.md in full (both versions), and the complete xtask/src/workflow_contract.rs diff plus the full lane-script test harness (LaneCase, run_lane, every a_lane_*/a_failing_channel_*/a_channel_whose_* test) and the complete queue_runner_references function with its test table, against base 07c93f32 → head 205d9c97 (4 commits). Checked the repository's root AGENTS.md (§3, §5, §6, §11, §12) — no violation found; the change's CI-only nature and lack of user-observable behaviour match the PR body's own checklist exemptions. Checked .github/workflows/*.yml's actual name: fields against the non-blocking finding above. Confirmed #950 is merged and is what this PR follows up on.

Verified directly by running the lane script against stubbed tools (no Rust/cargo/shellcheck toolchain available in this sandbox): a second-channel preflight failure (confirms the core fix) and a first-call preflight failure (confirms the pre-existing, intentionally-unified "set -e" behaviour described above). Did not run cargo test -p xtask / cargo clippy / shellcheck — no toolchain available; the PR's verification section claims these pass, not independently re-executed. CI: read via get_check_runs at this exact head — all 39 checks green, including Windows build and test (debug), which a review at an earlier commit on this PR noted as red (unrelated flake); it has since gone green, and the two skipped checks (Skill checks (skillscope), Sphinx docs build (-W)) are expected skips for this change.

Not reconciled against this PR's existing review discussion: per the review method, only login/commit metadata from prior reviews was checked (to confirm no duplicate review already existed at this exact head), never their content, and every finding above was reached independently of them.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@tomastola
tomastola added this pull request to the merge queue Nov 11, 2026
Merged via the queue into main with commit 8027e69 Nov 11, 2026
40 checks passed
@tomastola
tomastola deleted the ci/queue-lane-preflight-per-channel branch November 11, 2026 10:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants