Repository navigation
ci: run the nightly GPU E2E lanes on a shared GPU queue - #950
Conversation
308c6ca to
0bf3db9
Compare
|
Every lane is green except |
jussielo-amd
left a comment
There was a problem hiding this comment.
Looks good as an advisory path. This was a static review; I did not run the tests or the lanes.
Both workflows have permissions: {}, pass the dispatched branch and the request id only through env:, pin the one action to a full SHA, upload under if: always(), and the lane script keeps the suite's exit status through its trap. The doc and the workflows agree on artifact names, flags and inputs. CI is green except E2E tests (W7900, current), which was still queued and which you say is re-running; please confirm it before merging.
Comments are inline above. One more, non-blocking: the Verification section says "475 pass (at the head, 308c6caa)", but that SHA is not among the PR's commits and the head is 0bf3db9, so please re-run at the head and quote that SHA.
🤖 by agenthub on AMD AgentHub
juhovainio
left a comment
There was a problem hiding this comment.
Looks good. I found nothing blocking: the action is pinned by SHA, ${{ inputs.* }} goes through env vars, permissions are empty, and the lane script's failure and retry paths are tested. Keeping it advisory is the right call given the persistent runner on a public repo.
A few non-blocking things, three of them inline:
- The wait is described as "5 h 20 min at most" / "about five hours", but the doc caps it at 15000 s (4 h 10 min).
- The doc says the runner refuses any job that isn't one of the two workflows on
main, while its own "Why" section says that needs org-level configuration. Worth saying where that is enforced. - rustup is installed via an unpinned
curl | shin the lane script. - The PR body's verification line cites head
308c6caa; the head is now0bf3db90, so it's stale. - Please confirm
/opt/queue/bin/queue-publishis intentionally a neutral path to show in public text.
This is a read of the diff, not a run: I haven't exercised the start-from-GitHub path (it needs the workflows on main), and the nightly GPU lane is still pending on runner availability.
Adds nightly-queue.yml, which publishes nightly GPU lanes run on a shared GPU queue, one lane per GPU model in its fleet. The run itself happens next to the queue; this workflow prints a finished run's lane matrix and inputs, uploads the lanes' files and passes or fails with its verdict. It is manual-only and gates nothing: the queue's runs do not yet use a lane image carrying the suite, and the shared runner does not yet make the published verdict trustworthy enough to gate on. The scheduled lanes in nightly.yml remain the nightly gate. The publishing job holds no secret and no queue credential, and asks for no permissions. Its runner is a persistent self-hosted machine that any job naming its label can run on, so the queue credential stays with the queue side. The runner carries the queue-report label alone, without GitHub's default self-hosted, OS and architecture labels, so a job reaches it only by naming it. A workflow_contract test holds the workflow to that: dispatch only, two allowed expressions, one action (upload-artifact, pinned to a SHA, no hidden files), no environment, runtime-token variable or permission, and no YAML shapes the check cannot read. It also refuses any other workflow on main that names the label or this workflow literally, so none lands on the runner or waits on this one by mistake. Workflows on other branches are out of its reach, which the doc states. docs/nightly-on-the-queue.md is the contract: what a lane must write to /out to be published, how each lane outcome is reported, what the workflow needs, and why it is built this way, including what the shared runner does not yet protect. ci-hardware-testing.md links to it. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Each lane of the queue nightly now checks out the commit its run resolved and runs a command from that checkout, so what a lane runs is defined here and changes through a pull request like any other test. scripts/run-e2e-lane.sh is that command: the self-hosted nightly job's build, pre-warm and suite, from a cold start, with nightly scenarios included. A lane has no toolchain or cache from an earlier run, so it installs the pinned Rust toolchain when cargo is missing. The suite's results are copied to /out on every way out, so a failing suite still reports which scenarios failed, and the lane keeps the suite's status. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
queue-run.yml runs on the queue-report runner, on dispatch with a branch to test or nightly on main. It asks the queue side for a run of that branch, waits for it, then publishes it as nightly-queue.yml does. It holds no secret and no permission: the queue side starts the run with its own credential and trusts the request because the runner runs only these two workflows from main. The credential contract test now covers both workflows; a schedule's cron line is the one place `*` is allowed. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
queue-run.yml now starts, waits for and publishes every run, the nightly included, so nightly-queue.yml no longer publishes a default "newest scheduled run". It republishes one run by its required request_id, for instance one whose queue-run.yml job was cancelled while the run went on. With no default, an old pass can never stand in for a night that did not run; that night shows as a missing scheduled job instead. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
A lane starts from a fresh container against the public Ubuntu archive, and while a mirror is mid-sync apt-get update fails for a few minutes; the lane then failed before building anything. Retry it as the WSL lane in e2e-selfhosted.yml does. The script also now passes the repository's shellcheck settings, which it did not. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
The contract tests checked that the queue workflows hold no credential, not that they do their job: a run step that ignored the dispatched branch, published some other run, or was allowed to fail quietly still passed. They now pin each publishing step and the input behind it, read as its own YAML block, the schedule, that a running job is never cancelled for a newer one, and that each workflow uploads, whatever the verdict, the artifact the doc names for it. The lane script is run against stubbed tools: it runs the preflight, build, pre-warm and full suite in that order with the environment the nightly job gives them, exits with the suite's status, keeps the suite's results and its status even when the copy fails, rides out a failing apt-get update, and on an image without cargo installs rustup and leaves the toolchain to the checkout. The doc named one artifact for both workflows; it now names each. It also says what replacing the existing nightly lanes would lose (the nightly channel, Windows and WSL), that GitHub keeps only one dispatch waiting, and that a signed report helps only if something other than the runner checks the signature. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
The queue side no longer finishes a run on its own: a run is finished when the job following it asks. So a run whose queue-run.yml job was cancelled is finished by nightly-queue.yml, once its lanes are done, before it is published. Finishing fetches and scans every lane's files, which the publisher allows 30 minutes, so the job's timeout goes from 15 to 45 minutes. The contract doc says so, and that the runner's calls to the queue side are now three (start, status, finish), none carrying a file. The workflows' commands are unchanged. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
A queue lane now runs the full suite on the release channel and then the nightly channel, as the GPU jobs in nightly.yml do: one build, each channel with its own runtimes and its own results under <channel>/results. Both channels always run, and the lane fails if either did. The queue's results are trusted the way the self-hosted runners' nightly results are: the same infrastructure and access, and neither signs what it reports. The doc no longer calls them advisory or caveats their integrity, and says that native Windows and WSL remain the gap: the queue runs Linux pods, so those lanes stay on their dedicated runners. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
0bf3db9 to
5c96505
Compare
… main-only restriction lives - The lane installs a pinned rustup-init checked against a recorded SHA-256 instead of piping an installer script into a shell. - The wait bound is stated once and from the real limit: lanes wait at most 15000 s (4 h 10 min), and the job limit leaves room to finish and publish. - The doc says the main-only restriction is a check on the runner, outside this repository, and keeps the runner treated as untrusted. - The doc gives one window for each of a lane's files and a finished run's results, and no longer shows the publish command's path. - The contract refuses an expression inside a run: script, so inputs reach the shell only through env:. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
Thanks both. Updated at
Please take another look. |
juhovainio
left a comment
There was a problem hiding this comment.
I went through this PR and found no blocking problems. The workflows keep expressions out of run: bodies, the action and rustup pins check out, and the lane-script tests assert both exit status and leftover files. There are a few things worth fixing before merge, mostly small; five are inline below.
Inline
queue-run.yml:20: the comment about gating contradicts the doc, which now treats these results as trusted.queue-run.yml:46/nightly-queue.yml:41: pass--start=/--request=so a value starting with-can't be read as a flag.run-e2e-lane.sh:46: a whitespace-onlyE2E_CHANNELSthat isn't plain spaces (a tab, a newline) passes validation and the lane exits 0 having run nothing.workflow_contract.rs: thegonelist pins removed wording rather than a property of the code.
PR body
- It mentions "the exit trap removed" in the lane script, but
scripts/run-e2e-lane.shhas notrap. Please fix the text so it matches the diff.
Questions
- The doc says "the worst a request can do is start a run of a branch already in this repository". A
queue-reportworkflow can also run any existing branch's lane script in the queue pods and read retained results that were never published. If pod isolation is what makes the statement true, please say so in the doc. - The lane writes results only into the queue artifact, not the
e2e-gpu-<channel>-reportartifacts thatnightly.ymlreads. Is that intended, i.e. additional signal rather than a replacement? - GitHub keeps one waiting job, so a scheduled run can replace a waiting dispatch. Is that acceptable?
Posting as request-changes so these get a reply or a fix; none of them is a regression in existing behavior.
…st can reach - run-e2e-lane.sh counts the words in E2E_CHANNELS, so a tab or newline alone is refused like an empty list; the test gains those cases. - queue-run.yml no longer cross-references a gating condition the doc dropped. - The doc says what the lane pods are given, and that the queue's results are an additional signal next to nightly.yml's report artifacts. - The contract no longer pins wording removed from the doc. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
Thanks for the review. Updated at Your questions:
|
juhovainio
left a comment
There was a problem hiding this comment.
I went through this PR from the diff alone and found nothing blocking. I didn't run the tests, clippy or shellcheck locally, so treat this as a read-through rather than a full validation.
A few smaller things are left as inline comments: duplicate channels are accepted even though unknown ones are refused, set -e is off inside run_channel, the both-channel lane time against the 15000 s wait is unmeasured, and the 330 min timeout isn't tied to the doc by any test. One more to confirm: HF_HOME/UV_CACHE_DIR are set only for the prewarm step, so please check that matches what nightly.yml does for the suite step.
Checked and fine: the upload-artifact pin matches v7.0.2 and the x86_64 rustup-init checksum matches the published one. The workflows take no credentials or permissions, and inputs only reach the shell through env:.
One thing for the owner to accept consciously: anyone with write access can dispatch the queue run on any branch, so containment relies on out-of-repo controls (the runner only running main workflows, pod isolation). The doc says this openly. Approving on the basis of a diff-only read; CI is still pending on some checks.
…d tie the queue timeout to the wait limit - run-e2e-lane.sh refuses a channel named twice, and returns a failed cleanup of the previous channel's results instead of running on with them. - A contract test ties queue-run.yml's timeout, its comment and the doc to the 15000 s wait limit. - The doc says the both-channel lane time is not measured on a queue lane. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
|
Thanks. On |
juhovainio
left a comment
There was a problem hiding this comment.
I went through this at the current head and found nothing blocking. The pinned rustup-init checksums match the published ones, the lane script mirrors the build flags, env and prewarm/suite order of the nightly workflow, and the contract tests cover the failure paths. CI is green.
A few non-blocking points, left as inline comments:
- The second channel on the same GPU gets no preflight or stray-serve reclaim after the first channel's suite.
- A dispatched branch name beginning with
-could be read as an option byqueue-publish. - The wait the nightly uses isn't stated, and the two-channel lane time is unmeasured, so an incomplete lane may be expected at first.
- The contract test's substring match on
queue-runis broad.
Worth confirming as accepted decisions: the security model depends on the runner restriction that lives outside this repo, and results go to public artifacts behind a regex scan the doc itself calls a backstop.
This reflects a read of the diff, not a run on a real queue lane.
| . "${HOME}/.cargo/env" | ||
| fi | ||
|
|
||
| bash .github/scripts/gpu-preflight.sh |
There was a problem hiding this comment.
The GPU preflight runs once, before the build, so the nightly channel (run second on the same GPU) gets no preflight or stray-serve reclaim after the release suite. In nightly.yml each channel leg runs both. Consider calling the preflight at the start of run_channel.
There was a problem hiding this comment.
Agreed, and it is a real gap: the second channel gets no preflight or stray-serve reclaim after the first channel's suite. It will be fixed in a follow-up PR (the preflight at the start of run_channel, with a test that it runs once per channel), so this PR is not held for it.
| # Empty when scheduled. | ||
| BRANCH: ${{ inputs.branch }} | ||
| # One line: the workflow contract forbids backslashes in this file. | ||
| run: /opt/queue/bin/queue-publish --out "${RUNNER_TEMP}/results" --start "${BRANCH:-main}" |
There was a problem hiding this comment.
${BRANCH:-main} comes from a dispatch input; if it starts with - it may be read as an option by queue-publish. Consider --start="${BRANCH:-main}" or validating the name.
There was a problem hiding this comment.
Left as is, for the reason in the earlier thread: #950 (comment). The publisher takes the next token literally and accepts only the space form, so --start= would be rejected as an unknown argument, and the queue side refuses a branch starting with -.
| `release` channel and then the `nightly` one (`E2E_CHANNELS`, default `release | ||
| nightly`), each with its own runtimes and its own suite run, so a lane takes | ||
| about twice as long as one channel would. That time has not been measured on a | ||
| queue lane from a cold start, so a lane that cannot finish both channels within |
There was a problem hiding this comment.
This says the two-channel lane time is unmeasured and the lane may end up incomplete within the wait. Could you state which wait the nightly actually uses (the only queue trial so far used 30 minutes with a trivial command), so readers can tell whether an incomplete lane is expected?
There was a problem hiding this comment.
The nightly runs use the queue side's default wait of 14400 s (4 h), under the 15000 s limit. The only queue trial so far used 30 minutes with a trivial command, so there is no measurement for two real channels. An incomplete lane is therefore possible on the first runs, and the first nightly will show whether 4 h is enough. The doc will state the default wait in the follow-up PR, together with the preflight change.
| /// label, the workflow's file stem, or its name. Compared with case, quotes | ||
| /// and every whitespace character (line breaks included) removed, so a name | ||
| /// split across lines still counts; comments do not. | ||
| fn queue_runner_references(workflow: &str, queue_workflow_name: &str) -> Vec<String> { |
There was a problem hiding this comment.
Bare substring matching of queue-run/nightly-queue across the other workflows would also trip on unrelated future text such as queue-runner. Consider matching on word boundaries so the failure points at a real reference.
There was a problem hiding this comment.
Agreed: matching on word boundaries makes a failure point at a real reference. I will make that change in the same follow-up PR; it does not affect this one.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 02b6bc6fbf39
Review — no blocking findings
Full review of the whole change (single-context — fan-out was attempted and unavailable; see "What this covered").
No ticket named. The PR body's claims about scope ("does not replace or change any existing lane", CI-only, no product-facing messages) match the diff.
Blocking
None
Non-blocking
- A dispatched branch/request id beginning with
-could be read as an option byqueue-publish—.github/workflows/queue-run.yml:46,.github/workflows/nightly-queue.yml:41
Both steps pass a free-textworkflow_dispatchinput (branch,request_id, no format constraint in this repo) as a bare, space-separated argument:--start "${BRANCH:-main}"/--request "${REQUEST_ID}". A value starting with-reaching an external argument parser as a bare token is the classic argument-injection shape, distinct from the shell/expression-injection risk the contract tests do guard against correctly (both values only ever reachqueue-publish's argv, never shell-interpreted text — verified by tracingassert_queue_workflow_holds_no_credentialinxtask/src/workflow_contract.rs).queue-publishitself is maintained privately and out of this repo's scope, so its actual argument-parsing behavior can't be confirmed from here.
Confidence 70 · logic · Fix:--start=.../--request=..., or a literal--before the value.
Decisions for the author
None
Positive signals
assert_queue_workflow_holds_no_credential(xtask/src/workflow_contract.rs:8746) is a genuine allowlist over expressions, pinned actions, permissions and triggers rather than a list of known-bad spellings, and it separately forbids backslashes, YAML anchors/aliases and explicit? keyentries so an escape can't smuggle a forbidden construct past the text-based parsing.scripts/run-e2e-lane.sh's channel validation (unknown channel, a channel named twice, and whitespace-onlyE2E_CHANNELSincluding a literal tab/newline) runs before anyapt-get/build/network call, anda_lane_runs_only_the_channels_it_is_asked_for_and_refuses_othersexercises exactly those edge cases ("\t","\n"," \t\n ").- The rustup bootstrap fetches over pinned TLS and checks the downloaded binary's SHA-256 before
chmod +x/execution, with a separate pin per architecture, rather than piping a remote script into a shell.
Deployment notes
- This adds a new persistent self-hosted GitHub Actions runner (label
queue-report) on a public repository. Its security model depends on a control outside this repository: the runner is supposed to refuse any job except these two workflows as they exist onmain. Nothing in this repo can verify that control before merge; both workflows do correctly carrypermissions: {}and no secrets as a compensating control regardless. docs/nightly-on-the-queue.mdis a new contract page; two contract tests keep its channel/artifact/timeout claims tied to the code, but some numeric claims about the queue side itself (two-week result retention, ~a day for an unfinished run's lane files) are asserted only in prose and can't be checked from this repository.
What this covered
DEGRADED REVIEW (single-context, not fanned out — see below).
Read completely: both new workflows (.github/workflows/queue-run.yml, .github/workflows/nightly-queue.yml), the new lane script (scripts/run-e2e-lane.sh), the new doc (docs/nightly-on-the-queue.md), the three new hunks in docs/ci-hardware-testing.md, the 2-line doc-comment change plus the full ~959-line new test section added to xtask/src/workflow_contract.rs (lines 8704–9662), and this repo's AGENTS.md. Cross-checked specific claims (HF_HOME/UV_CACHE_DIR handling for the prewarm vs. suite step, the apt-get retry pattern, build flags) against the relevant jobs in the pre-existing nightly.yml/e2e-selfhosted.yml by targeted search rather than reading those ~2,000/~1,800-line files in full, since neither is itself in the diff — confirmed the lane script's HF_HOME/UV_CACHE_DIR handling genuinely mirrors nightly.yml's own e2e-gpu-nightly job (same pattern there too), so that is not a defect. Agent-instruction-adherence ran against AGENTS.md (§2 public-content/neutral-naming, §6 Actions SHA-pinning, the module-size cap's test-gated exemption) — no violation found. Code-comments pass ran as part of the above read, including tracing the set -e-suspended-by-|| comment in run_channel against actual bash semantics — comments match the code. Diffed 17f655d3f9…02b6bc6fbf39.
Did not run: fan-out was attempted (3 parallel workers covering workflows+script, the new contract-test section, and docs-vs-code) but every spawn failed immediately on a subagent concurrency limit already saturated by other sessions' agents, not this review's own, and retrying was advised against — so this is a single-context pass per the review method's degraded-fallback, and likely has lower coverage than a fanned-out pass would. History and prior-changes passes did not run: every touched file is either new or has a small, self-contained edit, with no separate pre-existing review history to check. Did not execute cargo test/cargo clippy/shellcheck against the PR's own content — per the review method an open PR is fetched read-only and never checked out, so verification here is by tracing the code against itself, not by running it. CI: all 29 checks on this commit are completed/success (clippy, the xtask/Rust jobs, and the hardware/E2E jobs included); nothing failing.
Not reconciled against this PR's own discussion: its existing review threads were incidentally visible while performing this task's mandatory author-identity claim-check (confirming no prior review from this reviewing identity already exists at this head) but were not read for, or used to shape, any finding above, and are not reconciled against this report.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
|
On the |
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. — not a bug fixdocs/architecture.md(the clap declaration incli.rsand the dispatch wiring inmain.rs/lib.rsare expected, not a violation). — no subcommand; CI onlyWhy
The nightly GPU lanes each hold a dedicated self-hosted runner for the whole job, so a GPU model can be tested only where a runner is registered for it. A shared queue of GPU workers can run one lane per GPU model it has, on whichever worker is free, and add new hardware without a workflow edit. This adds that path next to the existing lanes. Its results are trusted the way the self-hosted runners' nightly results are: the queue runs on the same infrastructure and under the same access, and neither signs what it reports. Native Windows and WSL are the remaining gap: the queue runs Linux pods, so those lanes stay on their dedicated runners.
What changes
queue-run.yml(dispatch with a branch, or nightly onmain) runs on one self-hosted runner, labelledqueue-report, that sits next to the queue. It asks the queue side for a run of that branch, waits for every lane, then prints the lane matrix and report, uploads the lanes' files asqueue-run-results, and passes or fails with the run's verdict.nightly-queue.ymlpublishes a run by its id, for one whosequeue-run.ymljob was cancelled: once the run's lanes are done, the queue side finishes it if no one has, then it is published as above.permissions: {}, use no secrets, environments or OIDC, and the runner holds no credential for the queue or the cluster. The runner carries only thequeue-reportlabel, without GitHub's default labels, so a job reaches it only by naming it, and it is set up to run only these two workflows as they are onmain. That restriction is a check on the runner, outside this repository, so nothing here can verify it and the runner is treated as untrusted regardless.scripts/run-e2e-lane.shis each lane's entrypoint, run from a checkout of the one commit the run resolved: it builds once from a cold start, then for each channel (release, thennightly, as the matrix innightly.ymldoes) pre-warms that channel's own runtimes, runs the full suite with nightly scenarios, and copies the suite's results to<channel>/resultswhatever the suite's status. Both channels always run, and the lane fails if either did. So what a lane runs is changed here, by a pull request.docs/nightly-on-the-queue.mdis the contract with the queue side: how to start and read a run, what a lane must write to be reported, what is never published, what the results are trusted for, and the remaining gap (native Windows and WSL).docs/ci-hardware-testing.mdpoints to it.xtask/src/workflow_contract.rs): the two workflows hold no credential (an allowlist over expressions, actions, permissions, environments and runtime-token variables, with YAML constructs that could hide a key refused outright); no other workflow names thequeue-reportlabel, the workflow files or their names; each publishing step passes the dispatched branch or the named run and cannot fail quietly; a running job is never cancelled for a newer one; norun:script contains an expression, so inputs reach the shell only throughenv:; and each workflow uploads, whatever the verdict, the artifact the doc names for it. The lane script runs against stubbed tools: preflight, build, pre-warm and the full suite in that order and environment, the suite's status kept even when copying its results fails,apt-get updateretried through a mirror mid-sync, rustup installed on an image without cargo from a pinned rustup-init checked against its recorded SHA-256 (a mismatch is never run, and each architecture has its own pin), both channels run in order with their own runtimes and results, a failing channel not hiding the other, and an unknown channel, a channel named twice, or a list with no channel in it (spaces, a tab or a newline), refused before anything is built. A doc test holds the channels and the Windows/WSL gap to what the script does.Verification
cargo test -p xtask: 715 pass (at the head,02b6bc6f);cargo clippy -p xtask --all-targets -- -D warningsclean; the lane script passes the repository's shellcheck settings.Each new contract test was checked against a deliberately broken input and fails on each: the branch dropped from
--start, the run id from--request, a default added torequest_id,continue-on-erroron the publishing step,cancel-in-progress: true, the upload'sif: always()turned intoif: success(), the doc naming the other artifact; in the lane script, the suite's status replaced by 0, a failed copy made fatal, the retry or its sleep removed, the preflight skipped, the pre-warm's cache paths or the build's features changed, nightly scenarios turned off, the rustup bootstrap altered. The new channel tests fail on the previous single-channel script.One full MI300X lane on the queue, from a cold start, with this branch's lane script: 299 of 306 scenarios passed. The seven failures are rocm-cli's own: six also fail in the MI300X lane of
nightly.yml, and one more shows only with an empty model cache.On the queue's runner, with a trivial lane command (2026-11-06): a run of
mainover 7 GPU models, 5 of which got a GPU and passed within its 30-minute wait; the 2 still waiting were cancelled at the deadline and reported with no result.queue-publish --request, the commandnightly-queue.ymlruns, finished and published it: matrix, report and 10 lane files, exit 0.Not verified: a run started from GitHub end to end. That needs these workflows on
main, and the runner runs only workflows frommain. Thenightlychannel has not yet run on a real queue lane: the channel loop is tested against stubbed tools only.Scope
queue-publishcommand) is maintained privately with the queue and is not part of this repository.