Skip to content

ci: fetch only the host's crates in the queue lane - #1057

Merged
tomastola merged 3 commits into
mainfrom
ci/queue-lane-host-only-fetch
Nov 12, 2026
Merged

tomastola merged 3 commits into
mainfrom
ci/queue-lane-host-only-fetch

Conversation

@tomastola

@tomastola tomastola commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to the review comments on #1009.

  • cargo fetch --locked now passes --target <host>, with the triple read from cargo -vV. The lane builds, pre-warms and runs the suite on the host and sets no build target, so the other platforms' crates were downloaded for nothing. A lane whose cargo reports no triple stops before the fetch.
  • The lane now also refuses to start (exit 1, before anything is installed) when CARGO_BUILD_TARGET is set in its environment, since a build target set from outside would need crates the host-only fetch skips. Anything that exports it into the lane's container would now fail the lane; the repo sets it nowhere. A target set through a cargo config is not covered.
  • Comment on the first cargo --version: it relies on rustup's proxy installing the toolchain named by rust-toolchain.toml on first call, and is a no-op where cargo is not rustup-managed.

Verified: into an empty CARGO_HOME, a host-only fetch downloads 386 MB, and cargo metadata --offline --locked --filter-platform <host> then resolves the whole workspace; the unrestricted fetch downloads 876 MB. cargo test -p xtask workflow_contract::tests::a_lane passes, including a new test for the no-triple case, and cargo fmt --check is clean. Not run: a real build offline, and shellcheck (not installed here). This is CI-internal, so no Gherkin scenario applies. The real lane runs only on a manual queue-run.yml dispatch or the nightly schedule, so the first dispatch after this merges is the real-world check.

`cargo fetch --locked` without `--target` downloads crates for every
platform. The lane builds, pre-warms and runs the suite on the host and
sets no build target, so fetch the host triple's crates only, reading
the triple from `cargo -vV`. Offline `cargo metadata` after a
host-only fetch into an empty CARGO_HOME resolves the full workspace
(386 MB downloaded, against 876 MB for the unrestricted fetch).

Also note that the first `cargo --version` relies on rustup's proxy
installing the toolchain named by rust-toolchain.toml, and is a no-op
where cargo is not rustup-managed.

Follow-up to review comments on #1009.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola requested a review from a team as a code owner November 12, 2026 10:18

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The host-only cargo fetch --target is a sensible scoping, it fails closed when cargo -vV reports no host triple (with a regression test), and the fake-cargo fixtures only answer the exact host-scoped fetch, so a revert to plain fetch --locked would fail the tests.

Non-blocking notes:

  • The "host-only is enough" claim rests on the offline cargo metadata check; no real offline build was run. If a later lane step ever builds for another target, a cold lane would hit the network without a retry (see inline).
  • Cosmetic: one over-long line left by the doc reflow (see inline).
  • The change is CI-internal, so no Gherkin scenario is needed; worth one line in the PR text saying so.

CI at this head is green for all completed checks; the pending GPU/E2E lanes are the known runner-pool outage.

Comment thread docs/nightly-on-the-queue.md Resolved
Comment thread scripts/run-e2e-lane.sh Resolved
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — no blocking findings

Full review of the whole change (3 files: scripts/run-e2e-lane.sh, xtask/src/workflow_contract.rs, docs/nightly-on-the-queue.md).
Follow-up to the review comments on #1009: that PR's two open threads (restrict the fetch to the host triple; document that the toolchain download relies on rustup's proxy) are both addressed here — confirmed against #1009's own review thread, where the author committed to this as a follow-up and later marked both threads "addressed in #1057."

Blocking

None

Non-blocking

  • A doc paragraph was rewrapped unevenly, leaving one line much longer than its neighbours — docs/nightly-on-the-queue.md:122
    The edited paragraph's lines are 73/77/107/79 characters; every other prose line nearby is 70–92 chars. This is the same class of nit a past review on this exact file/script pair caught and the author fixed by rewrapping evenly (see the review history on #999, where a reviewer wrote "the rewrap leaves line 112 very long; worth rewrapping evenly" and the paragraph was reflowed in response). Nothing lints this automatically.
    Confidence 80 · mechanical · Fix: reflow the paragraph evenly at roughly the same width as the surrounding lines.
  • The new host-triple plumbing is only ever exercised with one hardcoded triple string — xtask/src/workflow_contract.rs:9814 (stub uname), :9832-9855 (stub cargo's -vV/fetch --target arms)
    Every test that exercises the new cargo -vV → --target <host> path (the default case and the new "no host line" test) uses LaneCase::default()'s machine: "x86_64", and the stub cargo's -vV output and the fetch --locked --target … match arm are both hardcoded to the literal x86_64-unknown-linux-gnu. The one existing test that varies machine (the_rustup_pin_follows_the_machine, pre-existing, for the rustup installer's triple selection) doesn't touch this path. So the harness proves the plumbing works for that one literal string, not that an arbitrary host string flows through correctly end to end. Low real-world risk, since the script's handling is just text capture-and-interpolate rather than architecture-specific logic, but it's a gap in what the new tests actually prove.
    Confidence 72 · mechanical · Fix (optional): parametrize the stub's -vV host line and the fetch match arm on case.machine/an explicit triple, and add or extend a test with a non-x86_64 triple.

Decisions for the author

None

Positive signals

  • Using a herestring (<<<"${cargo_info}") rather than piping into the while read loop is the correct, deliberate choice — it avoids the classic bash bug where a piped while read runs in a subshell and loses the variable it's trying to set.
  • The missing-host-triple case fails fast with a clear message and a dedicated test (a_lane_stops_when_cargo_reports_no_host_triple) that asserts no fetch, no build, and no output directory — rather than silently passing an empty --target.
  • Leaving cargo -vV outside the retry wrapper is well-reasoned and matches retry's stated scope (network-reaching commands only): -vV is a local introspection call, and by the time it runs the toolchain is already installed via the preceding, already-retried cargo --version.

Deployment notes

As of this review, two of the self-hosted E2E checks — E2E tests (MI300X, current) and E2E tests (W7900, current) — were still queued and had not completed, even though every other check (including the Strix Halo self-hosted lanes, clippy, the xtask test job, and CodeQL) was green. These are the actual hardware lanes this script change touches, so it's worth confirming they finish green before merging rather than relying only on the checks that had completed at review time.

What this covered

Diff between origin/main's merge-base 61fdb25f and head 96f6275e. scripts/run-e2e-lane.sh and docs/nightly-on-the-queue.md read in full (both versions). In xtask/src/workflow_contract.rs (10,367 lines, almost entirely a #[cfg(test)] mod tests block): the full LaneCase struct and Default impl, the complete fake-executable/fixture generation in run_lane, and every test function exercising LaneCase/run_lane were read in full; the remaining unrelated workflow-YAML contract tests earlier in the file were not read line by line (confirmed by grep that no other location references the touched fields or command strings). Checked the repo's root AGENTS.md in full: the docs/architecture.md-entry and module-size rules don't apply (no file added/renamed/removed; the file is exempt as a #[cfg(test)]-gated module), and the Gherkin-scenario requirement doesn't apply (CI-only plumbing, no user-observable CLI behavior changes). Checked git history and past reviews on PRs #950, #999, and #1009 touching these same files for still-applicable concerns. Confirmed the workspace genuinely has per-platform (cfg(windows)/cfg(target_os = "macos")) dependency sections in several crates, and no .cargo/config.toml build-target override, so restricting the fetch to the host triple cannot starve the later untargeted build. CI checked via the live status rollup (see Deployment notes for the one caveat). Did not run cargo test/clippy/shellcheck directly in this environment (no Rust toolchain available here); relied on reading the code directly plus CI's own green checks and the author's stated verification.

Not reconciled against this PR's own review discussion — only commit-metadata was checked beforehand, to confirm no duplicate review already existed at this head.

The lane fetches only the host's crates, so a CARGO_BUILD_TARGET set from
outside would need crates it skipped and meet the network without a
retry. Refuse it up front, and exercise the host-scoped fetch with a
non-x86_64 triple. Rewrap the paragraph the earlier reflow left uneven.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Clippy's format_in_format_args refused the nested format! introduced with
the host-triple stub.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head 2838b5a6

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (61fdb25f…2838b5a6, 3 files: scripts/run-e2e-lane.sh, docs/nightly-on-the-queue.md, xtask/src/workflow_contract.rs).
Follow-up to the review comments on PR #1009; no tracker ticket is named by this PR.

Blocking

  • The PR description doesn't mention the new hard refusal on a CARGO_BUILD_TARGET environment variable — scripts/run-e2e-lane.sh:33-38, docs/nightly-on-the-queue.md:125, PR description
    The diff adds a real new failure mode: the lane now exits 1 immediately, before anything else runs, if CARGO_BUILD_TARGET is already set in its environment ("CARGO_BUILD_TARGET is set; unset it: the lane fetches only the host's crates"), and a dedicated test (a_lane_refuses_a_build_target_and_scopes_the_fetch_to_the_reported_host) exercises it. The PR's own description only says the lane "sets no build target, so the other platforms' crates were downloaded for nothing" and that "a future step that sets a build target would need its own fetch" — it never states that the lane will now refuse to start at all if that variable happens to already be set for an unrelated reason (e.g. something upstream in the container image). A reader going by the description alone would not learn this guard exists.
    Confidence 78 · mechanical · Fix: add one sentence to the PR description noting the lane now refuses to start when CARGO_BUILD_TARGET is set, matching what the code, the doc, and the second commit's own message already say — no code change needed.

Non-blocking

None

Decisions for the author

None

Positive signals

  • Using a herestring (<<<"${cargo_info}") instead of piping into the while read loop is correct and deliberate — confirmed by direct testing that bash's set -e does abort a bare var="$(cmd)" assignment on a failing command substitution, with the failing command's own exit status, which is exactly what the adjacent comment claims about an unreachable/broken cargo -vV.
  • The restriction to cargo fetch --locked --target <host> is consistent with the rest of the lane: cargo build is never given an explicit --target, so it already builds for the host implicitly, and nothing downstream (cargo xtask e2e-prewarm/e2e) needs a different target.
  • Test coverage for the new host-triple and build-target logic held up under hand-tracing of several plausible single-line mutations of the new bash conditionals (inverted test, dropped exit, wrong status) — the existing assertions already catch all of them.
  • The doc paragraph this change edited is evenly wrapped (72–78 characters per line, matching its neighbours), and the host-triple stub is exercised with a non-x86_64 triple (aarch64-unknown-linux-gnu) as well as the default — both already tightened within this PR's own commit history before this head, so there's nothing further to ask for on either front.

Deployment notes

  • One CI job, Security scan / Post PR findings, is red on this head, but its own log shows the cause is a transient GitHub API GET .../pulls/1057 returned HTTP 503 — an infrastructure flake unrelated to this diff, not a scanner finding (the actual scanners — CodeQL, Trivy, Bandit, Zizmor, Gitleaks — all passed on this head). Worth a rerun before merging rather than treating it as a real finding.
  • This PR's own CI does not exercise the real scripts/run-e2e-lane.sh end-to-end on hardware: the queue lane only runs via a manual queue-run.yml dispatch or the nightly schedule, not per-PR. The host-scoped fetch's real-world effect is validated here only by the xtask contract tests and the author's described manual local verification; worth confirming the first real dispatch after this merges completes its fetch and build cleanly.

What this covered

Diff between base 61fdb25f7a494f566d1d64de891315673f831595 and head 2838b5a625b212bb4611444c3e80b73dde85cfe3, fetched read-only by commit SHA into a scratch repository (the PR branch was never checked out). All three changed files read in full at the head commit: scripts/run-e2e-lane.sh, docs/nightly-on-the-queue.md, and the complete LaneCase/run_lane/lane-test block of xtask/src/workflow_contract.rs (the remaining ~10,000 lines of that file are unrelated workflow-YAML contract tests, confirmed by grep that none reference the touched fields or command strings). Checked the repository's root AGENTS.md in full; none of its guardrails (module-size, architecture-doc entries, Gherkin-scenario requirement, subprocess/FFI checklist) apply to a CI-internal shell/test-only change with no new source file and no user-observable CLI behavior. Walked the file history for all three files from the head commit back to the base: this PR is exactly three commits on top of 61fdb25f (#1009); the two earlier commits in this PR already fixed an unevenly-wrapped doc paragraph and broadened the host-triple test to a second, non-x86_64 machine value, both confirmed directly by reading the current file content rather than taken on anyone's claim. Confirmed no .cargo/config.toml or other in-repo mechanism sets a build target outside the CARGO_BUILD_TARGET environment variable the new check covers. Ran two independent fan-out passes as fresh workers (one for the script+doc, one for the Rust test module), each given only the file contents and no PR description or discussion; one candidate finding from the second pass (a minor test-assertion asymmetry between two sibling tests) was checked against the source and dropped after tracing that it does not survive any realistic single-line mutation of the guarded code — the existing assertions already catch those. Checked the live CI status rollup for the head commit (39 check runs) and read the one failing job's own log directly (see Deployment notes); did not re-derive what any passing check covers from its workflow definition. Did not run cargo test/clippy/shellcheck directly (no Rust toolchain available in this environment); relied on reading the code, hand-tracing the bash control flow (including one live local bash -c experiment confirming set -e's behavior on a failing command substitution), and the passing CI jobs.

Disclosure: while performing the mandatory pre-publish check for an existing review from this same account at this head, the API response also surfaced the full body of a pre-existing review already on this PR, left at an earlier commit in this same chain. That could not be avoided given the tool's response shape. Findings above were independently re-derived and verified against the current source rather than taken from that text, and both fan-out passes were run by workers who never saw it — but the consolidation and Step-9 judgement were made by a context that had incidentally seen it, which should be weighed accordingly. Not otherwise reconciled against this PR's own review discussion.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@tomastola

Copy link
Copy Markdown
Collaborator Author

Thanks for the review of 2838b5a. One blocking finding, and it held up:

  • Fixed: the PR description did not say the lane now refuses to start when CARGO_BUILD_TARGET is set. It does now, including that a target set through a cargo config is not covered, plus the one line about no Gherkin scenario applying. No code change was needed.
  • Deployment note, not changed: Security scan / Post PR findings failed on a transient HTTP 503 from the GitHub API, not on a finding. I have not re-run it, because a re-run spends CI on a repository I do not own. A maintainer re-run would clear it.
  • Deployment note, nothing to change: the real lane only runs on a manual queue-run.yml dispatch or the nightly schedule, so the first dispatch after this merges is the real check. I did not run it offline end to end either; the offline cargo metadata check is the evidence for the host-only fetch.

@tomastola
tomastola requested a review from juhovainio November 12, 2026 14:42

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I read the full diff and found nothing blocking. The host triple comes from cargo -vV rather than being hard-coded, the no-triple case stops before any fetch, and the tests cover a non-x86_64 host. The docs are updated alongside the script.

Two small non-blocking notes, one inline and one here:

  • The refusal on CARGO_BUILD_TARGET is described as happening before anything is installed, but the test doesn't assert that the apt/curl/preflight stubs never ran. Worth adding so the claim is held by a test.
  • Only the env var is guarded. A build.target in a cargo config would still need crates the host-only fetch skips. The script comment and PR body say so, but docs/nightly-on-the-queue.md doesn't; a short "(environment variable only)" note would close that.

One thing I couldn't confirm from the diff: whether anything in the lane runs an unfiltered cargo metadata offline, which could now miss other-platform manifests. The first real queue-lane dispatch will show it. This is a diff-only read; no lane run was exercised.

Comment thread scripts/run-e2e-lane.sh
channels="${E2E_CHANNELS:-release nightly}"

# The lane fetches only the host's crates, so a build target set from outside
# would need crates it skips and meet the network without a retry. Refused

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment and PR body say the refusal happens before anything is installed, but the new test doesn't assert that no apt/curl/preflight stub ran. Consider asserting it so the claim is held by a test.

Comment thread scripts/run-e2e-lane.sh
retry 4 30 cargo --version
retry 4 30 cargo fetch --locked
# Captured first, so a cargo that fails here aborts the lane with its own error.
cargo_info="$(cargo -vV)"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: cargo -vV isn't retried and aborts the lane on failure. That's fine since cargo --version just succeeded, and it matches the comment above.

@tomastola
tomastola added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 74f094b Nov 12, 2026
49 of 50 checks passed
@tomastola
tomastola deleted the ci/queue-lane-host-only-fetch branch November 12, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants