Skip to content

ci: run the nightly GPU E2E lanes on a shared GPU queue - #950

Merged
tomastola merged 11 commits into
mainfrom
ci/nightly-queue
Nov 9, 2026
Merged

tomastola merged 11 commits into
mainfrom
ci/nightly-queue

Conversation

@tomastola

@tomastola tomastola commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator
  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. — not a bug fix
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md (the clap declaration in cli.rs and the dispatch wiring in main.rs/lib.rs are expected, not a violation). — no subcommand; CI only
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will not happen. — no product messages change

Why

The nightly GPU lanes each hold a dedicated self-hosted runner for the whole job, so a GPU model can be tested only where a runner is registered for it. A shared queue of GPU workers can run one lane per GPU model it has, on whichever worker is free, and add new hardware without a workflow edit. This adds that path next to the existing lanes. Its results are trusted the way the self-hosted runners' nightly results are: the queue runs on the same infrastructure and under the same access, and neither signs what it reports. Native Windows and WSL are the remaining gap: the queue runs Linux pods, so those lanes stay on their dedicated runners.

What changes

  • queue-run.yml (dispatch with a branch, or nightly on main) runs on one self-hosted runner, labelled queue-report, that sits next to the queue. It asks the queue side for a run of that branch, waits for every lane, then prints the lane matrix and report, uploads the lanes' files as queue-run-results, and passes or fails with the run's verdict. nightly-queue.yml publishes a run by its id, for one whose queue-run.yml job was cancelled: once the run's lanes are done, the queue side finishes it if no one has, then it is published as above.
  • No credential and no permission on the runner. Both workflows have permissions: {}, use no secrets, environments or OIDC, and the runner holds no credential for the queue or the cluster. The runner carries only the queue-report label, without GitHub's default labels, so a job reaches it only by naming it, and it is set up to run only these two workflows as they are on main. That restriction is a check on the runner, outside this repository, so nothing here can verify it and the runner is treated as untrusted regardless.
  • scripts/run-e2e-lane.sh is each lane's entrypoint, run from a checkout of the one commit the run resolved: it builds once from a cold start, then for each channel (release, then nightly, as the matrix in nightly.yml does) pre-warms that channel's own runtimes, runs the full suite with nightly scenarios, and copies the suite's results to <channel>/results whatever the suite's status. Both channels always run, and the lane fails if either did. So what a lane runs is changed here, by a pull request.
  • docs/nightly-on-the-queue.md is the contract with the queue side: how to start and read a run, what a lane must write to be reported, what is never published, what the results are trusted for, and the remaining gap (native Windows and WSL). docs/ci-hardware-testing.md points to it.
  • Contract tests (xtask/src/workflow_contract.rs): the two workflows hold no credential (an allowlist over expressions, actions, permissions, environments and runtime-token variables, with YAML constructs that could hide a key refused outright); no other workflow names the queue-report label, the workflow files or their names; each publishing step passes the dispatched branch or the named run and cannot fail quietly; a running job is never cancelled for a newer one; no run: script contains an expression, so inputs reach the shell only through env:; and each workflow uploads, whatever the verdict, the artifact the doc names for it. The lane script runs against stubbed tools: preflight, build, pre-warm and the full suite in that order and environment, the suite's status kept even when copying its results fails, apt-get update retried through a mirror mid-sync, rustup installed on an image without cargo from a pinned rustup-init checked against its recorded SHA-256 (a mismatch is never run, and each architecture has its own pin), both channels run in order with their own runtimes and results, a failing channel not hiding the other, and an unknown channel, a channel named twice, or a list with no channel in it (spaces, a tab or a newline), refused before anything is built. A doc test holds the channels and the Windows/WSL gap to what the script does.

Verification

  • cargo test -p xtask: 715 pass (at the head, 02b6bc6f); cargo clippy -p xtask --all-targets -- -D warnings clean; the lane script passes the repository's shellcheck settings.

  • Each new contract test was checked against a deliberately broken input and fails on each: the branch dropped from --start, the run id from --request, a default added to request_id, continue-on-error on the publishing step, cancel-in-progress: true, the upload's if: always() turned into if: success(), the doc naming the other artifact; in the lane script, the suite's status replaced by 0, a failed copy made fatal, the retry or its sleep removed, the preflight skipped, the pre-warm's cache paths or the build's features changed, nightly scenarios turned off, the rustup bootstrap altered. The new channel tests fail on the previous single-channel script.

  • One full MI300X lane on the queue, from a cold start, with this branch's lane script: 299 of 306 scenarios passed. The seven failures are rocm-cli's own: six also fail in the MI300X lane of nightly.yml, and one more shows only with an empty model cache.

  • On the queue's runner, with a trivial lane command (2026-11-06): a run of main over 7 GPU models, 5 of which got a GPU and passed within its 30-minute wait; the 2 still waiting were cancelled at the deadline and reported with no result. queue-publish --request, the command nightly-queue.yml runs, finished and published it: matrix, report and 10 lane files, exit 0.

Not verified: a run started from GitHub end to end. That needs these workflows on main, and the runner runs only workflows from main. The nightly channel has not yet run on a real queue lane: the channel loop is tested against stubbed tools only.

Scope

  • Does not replace or change any existing lane, and changes no required check. Native Windows and WSL are the remaining gap: the queue runs Linux pods, so those lanes stay on their dedicated runners; the doc says so.
  • The queue side (starting runs, submitting lanes, collecting and scanning results, and the queue-publish command) is maintained privately with the queue and is not part of this repository.

@tomastola
tomastola marked this pull request as ready for review November 9, 2026 10:46
@tomastola
tomastola requested a review from a team as a code owner November 9, 2026 10:46
@tomastola
tomastola requested a review from johnl-amd November 9, 2026 10:46
@tomastola

Copy link
Copy Markdown
Collaborator Author

Every lane is green except E2E tests (W7900, current), which is re-running on a busy runner. The consolidated report re-runs after it.

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good as an advisory path. This was a static review; I did not run the tests or the lanes.

Both workflows have permissions: {}, pass the dispatched branch and the request id only through env:, pin the one action to a full SHA, upload under if: always(), and the lane script keeps the suite's exit status through its trap. The doc and the workflows agree on artifact names, flags and inputs. CI is green except E2E tests (W7900, current), which was still queued and which you say is re-running; please confirm it before merging.

Comments are inline above. One more, non-blocking: the Verification section says "475 pass (at the head, 308c6caa)", but that SHA is not among the PR's commits and the head is 0bf3db9, so please re-run at the head and quote that SHA.


🤖 by agenthub on AMD AgentHub

Comment thread xtask/src/workflow_contract.rs Resolved
Comment thread docs/nightly-on-the-queue.md Resolved
Comment thread docs/nightly-on-the-queue.md Resolved
Comment thread .github/workflows/queue-run.yml Resolved

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I found nothing blocking: the action is pinned by SHA, ${{ inputs.* }} goes through env vars, permissions are empty, and the lane script's failure and retry paths are tested. Keeping it advisory is the right call given the persistent runner on a public repo.

A few non-blocking things, three of them inline:

  • The wait is described as "5 h 20 min at most" / "about five hours", but the doc caps it at 15000 s (4 h 10 min).
  • The doc says the runner refuses any job that isn't one of the two workflows on main, while its own "Why" section says that needs org-level configuration. Worth saying where that is enforced.
  • rustup is installed via an unpinned curl | sh in the lane script.
  • The PR body's verification line cites head 308c6caa; the head is now 0bf3db90, so it's stale.
  • Please confirm /opt/queue/bin/queue-publish is intentionally a neutral path to show in public text.

This is a read of the diff, not a run: I haven't exercised the start-from-GitHub path (it needs the workflows on main), and the nightly GPU lane is still pending on runner availability.

Comment thread .github/workflows/queue-run.yml Resolved
Comment thread docs/nightly-on-the-queue.md Resolved
Comment thread scripts/run-e2e-lane.sh Resolved
Adds nightly-queue.yml, which publishes nightly GPU lanes run on a shared
GPU queue, one lane per GPU model in its fleet. The run itself happens next to
the queue; this workflow prints a finished run's lane matrix and inputs,
uploads the lanes' files and passes or fails with its verdict. It is
manual-only and gates nothing: the queue's runs do not yet use a lane image
carrying the suite, and the shared runner does not yet make the published
verdict trustworthy enough to gate on. The scheduled lanes in nightly.yml
remain the nightly gate.

The publishing job holds no secret and no queue credential, and asks for no
permissions. Its runner is a persistent self-hosted machine that any job
naming its label can run on, so the queue credential stays with the queue
side. The runner carries the queue-report label alone, without GitHub's
default self-hosted, OS and architecture labels, so a job reaches it only by
naming it. A workflow_contract test holds the workflow to that: dispatch only,
two allowed expressions, one action (upload-artifact, pinned to a SHA, no
hidden files), no environment, runtime-token variable or permission, and no
YAML shapes the check cannot read. It also refuses any other workflow on main
that names the label or this workflow literally, so none lands on the runner
or waits on this one by mistake. Workflows on other branches are out of its reach,
which the doc states.

docs/nightly-on-the-queue.md is the contract: what a lane must write to /out
to be published, how each lane outcome is reported, what the workflow needs,
and why it is built this way, including what the shared runner does not yet
protect. ci-hardware-testing.md links to it.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Each lane of the queue nightly now checks out the commit its run
resolved and runs a command from that checkout, so what a lane runs is
defined here and changes through a pull request like any other test.

scripts/run-e2e-lane.sh is that command: the self-hosted nightly job's
build, pre-warm and suite, from a cold start, with nightly scenarios
included. A lane has no toolchain or cache from an earlier run, so it
installs the pinned Rust toolchain when cargo is missing. The suite's
results are copied to /out on every way out, so a failing suite still
reports which scenarios failed, and the lane keeps the suite's status.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
queue-run.yml runs on the queue-report runner, on dispatch with a branch
to test or nightly on main. It asks the queue side for a run of that
branch, waits for it, then publishes it as nightly-queue.yml does. It
holds no secret and no permission: the queue side starts the run with its
own credential and trusts the request because the runner runs only these
two workflows from main.

The credential contract test now covers both workflows; a schedule's
cron line is the one place `*` is allowed.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
queue-run.yml now starts, waits for and publishes every run, the nightly
included, so nightly-queue.yml no longer publishes a default "newest
scheduled run". It republishes one run by its required request_id, for
instance one whose queue-run.yml job was cancelled while the run went on.
With no default, an old pass can never stand in for a night that did not
run; that night shows as a missing scheduled job instead.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
A lane starts from a fresh container against the public Ubuntu archive,
and while a mirror is mid-sync apt-get update fails for a few minutes;
the lane then failed before building anything. Retry it as the WSL lane
in e2e-selfhosted.yml does. The script also now passes the repository's
shellcheck settings, which it did not.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
The contract tests checked that the queue workflows hold no credential,
not that they do their job: a run step that ignored the dispatched
branch, published some other run, or was allowed to fail quietly still
passed. They now pin each publishing step and the input behind it, read
as its own YAML block, the schedule, that a running job is never
cancelled for a newer one, and that each workflow uploads, whatever the
verdict, the artifact the doc names for it. The lane script is run
against stubbed tools: it runs the preflight, build, pre-warm and full
suite in that order with the environment the nightly job gives them,
exits with the suite's status, keeps the suite's results and its status
even when the copy fails, rides out a failing apt-get update, and on an
image without cargo installs rustup and leaves the toolchain to the
checkout.

The doc named one artifact for both workflows; it now names each. It
also says what replacing the existing nightly lanes would lose (the
nightly channel, Windows and WSL), that GitHub keeps only one dispatch
waiting, and that a signed report helps only if something other than
the runner checks the signature.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
The queue side no longer finishes a run on its own: a run is finished
when the job following it asks. So a run whose queue-run.yml job was
cancelled is finished by nightly-queue.yml, once its lanes are done,
before it is published. Finishing fetches and scans every lane's files,
which the publisher allows 30 minutes, so the job's timeout goes from 15
to 45 minutes.

The contract doc says so, and that the runner's calls to the queue side
are now three (start, status, finish), none carrying a file. The
workflows' commands are unchanged.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
A queue lane now runs the full suite on the release channel and then the
nightly channel, as the GPU jobs in nightly.yml do: one build, each channel with
its own runtimes and its own results under <channel>/results. Both channels
always run, and the lane fails if either did.

The queue's results are trusted the way the self-hosted runners' nightly
results are: the same infrastructure and access, and neither signs what it
reports. The doc no longer calls them advisory or caveats their integrity, and
says that native Windows and WSL remain the gap: the queue runs Linux pods, so
those lanes stay on their dedicated runners.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola tomastola changed the title ci: run the nightly GPU E2E lanes on a shared GPU queue (advisory) ci: run the nightly GPU E2E lanes on a shared GPU queue Nov 9, 2026
… main-only restriction lives

- The lane installs a pinned rustup-init checked against a recorded SHA-256
  instead of piping an installer script into a shell.
- The wait bound is stated once and from the real limit: lanes wait at most
  15000 s (4 h 10 min), and the job limit leaves room to finish and publish.
- The doc says the main-only restriction is a check on the runner, outside this
  repository, and keeps the runner treated as untrusted.
- The doc gives one window for each of a lane's files and a finished run's
  results, and no longer shows the publish command's path.
- The contract refuses an expression inside a run: script, so inputs reach the
  shell only through env:.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola

Copy link
Copy Markdown
Collaborator Author

Thanks both. Updated at 447bfc0d:

  • @juhovainio: the lanes are not made a required check, so dropping the word "advisory" does not change what can block a merge. The title and doc no longer use it.
  • The queue lanes now run both the release and the nightly channel. Native Windows and WSL stay on their dedicated runners, because the queue runs Linux pods; the doc says so.
  • Fixed the wait figure (4 h 10 min), the main-only wording, and the rustup install (pinned and checksummed). The PR text and doc no longer name the publish command's path; only the two run: lines still call it.
  • The head in the description is refreshed, and the full suite was re-run at it.

Please take another look.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this PR and found no blocking problems. The workflows keep expressions out of run: bodies, the action and rustup pins check out, and the lane-script tests assert both exit status and leftover files. There are a few things worth fixing before merge, mostly small; five are inline below.

Inline

  • queue-run.yml:20: the comment about gating contradicts the doc, which now treats these results as trusted.
  • queue-run.yml:46 / nightly-queue.yml:41: pass --start= / --request= so a value starting with - can't be read as a flag.
  • run-e2e-lane.sh:46: a whitespace-only E2E_CHANNELS that isn't plain spaces (a tab, a newline) passes validation and the lane exits 0 having run nothing.
  • workflow_contract.rs: the gone list pins removed wording rather than a property of the code.

PR body

  • It mentions "the exit trap removed" in the lane script, but scripts/run-e2e-lane.sh has no trap. Please fix the text so it matches the diff.

Questions

  • The doc says "the worst a request can do is start a run of a branch already in this repository". A queue-report workflow can also run any existing branch's lane script in the queue pods and read retained results that were never published. If pod isolation is what makes the statement true, please say so in the doc.
  • The lane writes results only into the queue artifact, not the e2e-gpu-<channel>-report artifacts that nightly.yml reads. Is that intended, i.e. additional signal rather than a replacement?
  • GitHub keeps one waiting job, so a scheduled run can replace a waiting dispatch. Is that acceptable?

Posting as request-changes so these get a reply or a fix; none of them is a regression in existing behavior.

Comment thread .github/workflows/queue-run.yml Resolved
Comment thread .github/workflows/queue-run.yml Resolved
Comment thread .github/workflows/nightly-queue.yml Resolved
Comment thread scripts/run-e2e-lane.sh Resolved
Comment thread xtask/src/workflow_contract.rs Resolved
…st can reach

- run-e2e-lane.sh counts the words in E2E_CHANNELS, so a tab or newline alone
  is refused like an empty list; the test gains those cases.
- queue-run.yml no longer cross-references a gating condition the doc dropped.
- The doc says what the lane pods are given, and that the queue's results are
  an additional signal next to nightly.yml's report artifacts.
- The contract no longer pins wording removed from the doc.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola

Copy link
Copy Markdown
Collaborator Author

Thanks for the review. Updated at 35802691; all five inline threads are answered, and the description no longer lists an exit trap (the script has none) and cites the new head.

Your questions:

  • What a request can reach. The doc said only that the worst a request can do is start a run of a branch of this repository. It now says more: that branch's lane command then runs in the lane pods. The lane spec gives those pods no queue or cluster credential and mounts neither the retained results nor the runner's files, so the command publishes only what it writes to /out, through the scan. What else a pod can reach is the queue's own isolation, which this repository cannot verify, and the doc says that instead of claiming it.
  • Results only in the queue artifact. Intended: they are an additional signal, not a replacement. The lanes write only into queue-run-results, and the e2e-gpu-<channel>-report artifacts that the consolidated report in nightly.yml reads are not produced from them. The doc now says so.
  • A schedule replacing a waiting dispatch. Accepted. GitHub keeps one waiting job, and the queue side refuses a second run while one is unfinished anyway, so there was never room for both. A dispatch replaced that way never runs, and the doc says so under "Starting a run"; re-dispatching it is the remedy.

@tomastola
tomastola requested a review from juhovainio November 9, 2026 13:10

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this PR from the diff alone and found nothing blocking. I didn't run the tests, clippy or shellcheck locally, so treat this as a read-through rather than a full validation.

A few smaller things are left as inline comments: duplicate channels are accepted even though unknown ones are refused, set -e is off inside run_channel, the both-channel lane time against the 15000 s wait is unmeasured, and the 330 min timeout isn't tied to the doc by any test. One more to confirm: HF_HOME/UV_CACHE_DIR are set only for the prewarm step, so please check that matches what nightly.yml does for the suite step.

Checked and fine: the upload-artifact pin matches v7.0.2 and the x86_64 rustup-init checksum matches the published one. The workflows take no credentials or permissions, and inputs only reach the shell through env:.

One thing for the owner to accept consciously: anyone with write access can dispatch the queue run on any branch, so containment relies on out-of-repo controls (the runner only running main workflows, pod isolation). The doc says this openly. Approving on the basis of a diff-only read; CI is still pending on some checks.

Comment thread scripts/run-e2e-lane.sh Resolved
Comment thread scripts/run-e2e-lane.sh Resolved
Comment thread docs/nightly-on-the-queue.md Resolved
Comment thread .github/workflows/queue-run.yml Resolved
…d tie the queue timeout to the wait limit

- run-e2e-lane.sh refuses a channel named twice, and returns a failed cleanup
  of the previous channel's results instead of running on with them.
- A contract test ties queue-run.yml's timeout, its comment and the doc to the
  15000 s wait limit.
- The doc says the both-channel lane time is not measured on a queue lane.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola

Copy link
Copy Markdown
Collaborator Author

Thanks. On HF_HOME and UV_CACHE_DIR: this matches nightly.yml, which sets them only on the pre-warm step (e2e-selfhosted.yml). The suite step sets neither; the harness derives both from E2E_SHARED_CACHE_DIR and E2E_SHARED_UV_CACHE_DIR (tests/e2e-cucumber/tests/e2e.rs), which the lane script exports for the whole run. So the suite sees the same caches as there. The four inline threads are answered in 02b6bc6f.

@tomastola
tomastola requested a review from juhovainio November 9, 2026 13:33

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this at the current head and found nothing blocking. The pinned rustup-init checksums match the published ones, the lane script mirrors the build flags, env and prewarm/suite order of the nightly workflow, and the contract tests cover the failure paths. CI is green.

A few non-blocking points, left as inline comments:

  • The second channel on the same GPU gets no preflight or stray-serve reclaim after the first channel's suite.
  • A dispatched branch name beginning with - could be read as an option by queue-publish.
  • The wait the nightly uses isn't stated, and the two-channel lane time is unmeasured, so an incomplete lane may be expected at first.
  • The contract test's substring match on queue-run is broad.

Worth confirming as accepted decisions: the security model depends on the runner restriction that lives outside this repo, and results go to public artifacts behind a regex scan the doc itself calls a backstop.

This reflects a read of the diff, not a run on a real queue lane.

Comment thread scripts/run-e2e-lane.sh
. "${HOME}/.cargo/env"
fi

bash .github/scripts/gpu-preflight.sh

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GPU preflight runs once, before the build, so the nightly channel (run second on the same GPU) gets no preflight or stray-serve reclaim after the release suite. In nightly.yml each channel leg runs both. Consider calling the preflight at the start of run_channel.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and it is a real gap: the second channel gets no preflight or stray-serve reclaim after the first channel's suite. It will be fixed in a follow-up PR (the preflight at the start of run_channel, with a test that it runs once per channel), so this PR is not held for it.

# Empty when scheduled.
BRANCH: ${{ inputs.branch }}
# One line: the workflow contract forbids backslashes in this file.
run: /opt/queue/bin/queue-publish --out "${RUNNER_TEMP}/results" --start "${BRANCH:-main}"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

${BRANCH:-main} comes from a dispatch input; if it starts with - it may be read as an option by queue-publish. Consider --start="${BRANCH:-main}" or validating the name.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left as is, for the reason in the earlier thread: #950 (comment). The publisher takes the next token literally and accepts only the space form, so --start= would be rejected as an unknown argument, and the queue side refuses a branch starting with -.

`release` channel and then the `nightly` one (`E2E_CHANNELS`, default `release
nightly`), each with its own runtimes and its own suite run, so a lane takes
about twice as long as one channel would. That time has not been measured on a
queue lane from a cold start, so a lane that cannot finish both channels within

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This says the two-channel lane time is unmeasured and the lane may end up incomplete within the wait. Could you state which wait the nightly actually uses (the only queue trial so far used 30 minutes with a trivial command), so readers can tell whether an incomplete lane is expected?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The nightly runs use the queue side's default wait of 14400 s (4 h), under the 15000 s limit. The only queue trial so far used 30 minutes with a trivial command, so there is no measurement for two real channels. An incomplete lane is therefore possible on the first runs, and the first nightly will show whether 4 h is enough. The doc will state the default wait in the follow-up PR, together with the preflight change.

/// label, the workflow's file stem, or its name. Compared with case, quotes
/// and every whitespace character (line breaks included) removed, so a name
/// split across lines still counts; comments do not.
fn queue_runner_references(workflow: &str, queue_workflow_name: &str) -> Vec<String> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bare substring matching of queue-run/nightly-queue across the other workflows would also trip on unrelated future text such as queue-runner. Consider matching on word boundaries so the failure points at a real reference.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: matching on word boundaries makes a failure point at a real reference. I will make that change in the same follow-up PR; it does not affect this one.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 02b6bc6fbf39

Review — no blocking findings

Full review of the whole change (single-context — fan-out was attempted and unavailable; see "What this covered").
No ticket named. The PR body's claims about scope ("does not replace or change any existing lane", CI-only, no product-facing messages) match the diff.

Blocking

None

Non-blocking

  • A dispatched branch/request id beginning with - could be read as an option by queue-publish — .github/workflows/queue-run.yml:46, .github/workflows/nightly-queue.yml:41
    Both steps pass a free-text workflow_dispatch input (branch, request_id, no format constraint in this repo) as a bare, space-separated argument: --start "${BRANCH:-main}" / --request "${REQUEST_ID}". A value starting with - reaching an external argument parser as a bare token is the classic argument-injection shape, distinct from the shell/expression-injection risk the contract tests do guard against correctly (both values only ever reach queue-publish's argv, never shell-interpreted text — verified by tracing assert_queue_workflow_holds_no_credential in xtask/src/workflow_contract.rs). queue-publish itself is maintained privately and out of this repo's scope, so its actual argument-parsing behavior can't be confirmed from here.
    Confidence 70 · logic · Fix: --start=... / --request=..., or a literal -- before the value.

Decisions for the author

None

Positive signals

  • assert_queue_workflow_holds_no_credential (xtask/src/workflow_contract.rs:8746) is a genuine allowlist over expressions, pinned actions, permissions and triggers rather than a list of known-bad spellings, and it separately forbids backslashes, YAML anchors/aliases and explicit ? key entries so an escape can't smuggle a forbidden construct past the text-based parsing.
  • scripts/run-e2e-lane.sh's channel validation (unknown channel, a channel named twice, and whitespace-only E2E_CHANNELS including a literal tab/newline) runs before any apt-get/build/network call, and a_lane_runs_only_the_channels_it_is_asked_for_and_refuses_others exercises exactly those edge cases ("\t", "\n", " \t\n ").
  • The rustup bootstrap fetches over pinned TLS and checks the downloaded binary's SHA-256 before chmod +x/execution, with a separate pin per architecture, rather than piping a remote script into a shell.

Deployment notes

  • This adds a new persistent self-hosted GitHub Actions runner (label queue-report) on a public repository. Its security model depends on a control outside this repository: the runner is supposed to refuse any job except these two workflows as they exist on main. Nothing in this repo can verify that control before merge; both workflows do correctly carry permissions: {} and no secrets as a compensating control regardless.
  • docs/nightly-on-the-queue.md is a new contract page; two contract tests keep its channel/artifact/timeout claims tied to the code, but some numeric claims about the queue side itself (two-week result retention, ~a day for an unfinished run's lane files) are asserted only in prose and can't be checked from this repository.

What this covered

DEGRADED REVIEW (single-context, not fanned out — see below).

Read completely: both new workflows (.github/workflows/queue-run.yml, .github/workflows/nightly-queue.yml), the new lane script (scripts/run-e2e-lane.sh), the new doc (docs/nightly-on-the-queue.md), the three new hunks in docs/ci-hardware-testing.md, the 2-line doc-comment change plus the full ~959-line new test section added to xtask/src/workflow_contract.rs (lines 8704–9662), and this repo's AGENTS.md. Cross-checked specific claims (HF_HOME/UV_CACHE_DIR handling for the prewarm vs. suite step, the apt-get retry pattern, build flags) against the relevant jobs in the pre-existing nightly.yml/e2e-selfhosted.yml by targeted search rather than reading those ~2,000/~1,800-line files in full, since neither is itself in the diff — confirmed the lane script's HF_HOME/UV_CACHE_DIR handling genuinely mirrors nightly.yml's own e2e-gpu-nightly job (same pattern there too), so that is not a defect. Agent-instruction-adherence ran against AGENTS.md (§2 public-content/neutral-naming, §6 Actions SHA-pinning, the module-size cap's test-gated exemption) — no violation found. Code-comments pass ran as part of the above read, including tracing the set -e-suspended-by-|| comment in run_channel against actual bash semantics — comments match the code. Diffed 17f655d3f9…02b6bc6fbf39.

Did not run: fan-out was attempted (3 parallel workers covering workflows+script, the new contract-test section, and docs-vs-code) but every spawn failed immediately on a subagent concurrency limit already saturated by other sessions' agents, not this review's own, and retrying was advised against — so this is a single-context pass per the review method's degraded-fallback, and likely has lower coverage than a fanned-out pass would. History and prior-changes passes did not run: every touched file is either new or has a small, self-contained edit, with no separate pre-existing review history to check. Did not execute cargo test/cargo clippy/shellcheck against the PR's own content — per the review method an open PR is fetched read-only and never checked out, so verification here is by tracing the code against itself, not by running it. CI: all 29 checks on this commit are completed/success (clippy, the xtask/Rust jobs, and the hardware/E2E jobs included); nothing failing.
Not reconciled against this PR's own discussion: its existing review threads were incidentally visible while performing this task's mandatory author-identity claim-check (confirming no prior review from this reviewing identity already exists at this head) but were not read for, or used to shape, any finding above, and are not reconciled against this report.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@tomastola

Copy link
Copy Markdown
Collaborator Author

On the --prefixed value in queue-run.yml and nightly-queue.yml: no change. The publisher takes the token after --start or --request literally and accepts only the space form, so --start=... would be rejected as an unknown argument. The queue side refuses a branch name starting with -, and a request id that is not an id fails. The reasoning is in my reply on the earlier thread: #950 (comment)

@tomastola
tomastola added this pull request to the merge queue Nov 9, 2026
Merged via the queue into main with commit 07c93f3 Nov 9, 2026
36 of 37 checks passed
@tomastola
tomastola deleted the ci/nightly-queue branch November 9, 2026 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants