Skip to content

ci(windows): run Windows debug and release builds as parallel jobs - #810

Merged
jussielo-amd merged 9 commits into
ROCm:mainfrom
jussielo-amd:ci/windows-parallel-jobs
Nov 2, 2026
Merged

jussielo-amd merged 9 commits into
ROCm:mainfrom
jussielo-amd:ci/windows-parallel-jobs

Conversation

@jussielo-amd

@jussielo-amd jussielo-amd commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The Windows build-and-test job is the longest pole of every CI run: 10 to 16 minutes on recent pushes to main, and 20 or more on pull requests from forks, which get no shared compiler cache. It also starts a couple of minutes late because it waits for the Linux clippy and prek jobs. This splits it into two parallel jobs and removes that wait.

Measured so far, on this pull request's cold-cache runs, the two jobs took about 14.7 minutes (windows-debug) and 20.5 to 21 minutes (windows-release), and the run reached its last non-GPU job after about 22 minutes. The gain on main, where the cache is warm, has not been measured since the split; the estimate from step timings is that Windows drops from about 10 to 16 minutes to roughly the longer of the two lanes, which is not a claim this pull request can verify. It should be checked over several push runs on main after merge, comparing the two jobs' durations and the time until the last non-GPU job finishes, before rebalancing steps between the lanes.

What changed

  • windows-debug: Windows clippy, dev build, workspace tests, local smoke, and the harness self-tests.
  • windows-release: release build, install lifecycle e2e, release-readiness self-test, installer syntax check, pinned-key check, and PR bundle packaging. None of these consume the debug side's output. The one thing this side gives up is the debug build's warm-up of the e2e harness's dev-profile dependencies, which it now compiles for itself.
  • windows-build-and-test keeps its name as the required check and becomes an aggregator that needs changes, windows-debug and windows-release, and requires each to have finished exactly success. A skip is accepted only for a manual dispatch that did not ask for binaries. The job has no name: of its own, so the check context branch protection matches is unchanged.
  • Whether the Windows jobs run is decided once, as a new changes.windows output. Both jobs gate on it and the aggregator derives its expected skip from it, so the two cannot disagree.
  • Neither Windows job waits for clippy or prek any more. Both remain required checks of their own, so a lint failure still blocks the merge.
  • Every build, test and check step of the old job is carried over unchanged into exactly one new job. The shared setup (checkout, sccache install and probe, Rust toolchain, sccache stats) runs in both. I checked this by diffing the parsed steps before and after.
  • Comments and docs that described the old single job are updated, including the timeout note, the changes job's description of its outputs, and the remote-control job's gating note.

Why the gate is not weakened

A skipped required check reads as passing, so the aggregator compares each result for exact equality instead of testing for failure: failure, cancelled and an unexpected skipped all fail it, and a failed changes fails it even when the work jobs are meant to be skipped. The contract tests in xtask/src/workflow_contract.rs pin this:

  • The aggregator must await every job in the Windows lane (found by runner, with an explicit allowlist for powershell-lint), bind each job's result to exactly one env variable, and check that variable in its script. The script is executed under bash -e over every combination of job results, using the variable names taken from the workflow's own bindings.
  • The aggregator job sets exactly runs-on, timeout-minutes, needs, if and steps, its if: is exactly !cancelled(), and it is one step with exactly the keys name, env and run. A name: would rename the required check, and an if, continue-on-error or shell would change when it runs or whether it can fail.
  • The work jobs' needs, runner and if: are pinned exactly with no job-level continue-on-error, and the two jobs' env and setup steps cannot drift apart.
  • Each gating step is pinned by name: it must exist once, contain its command, carry its exact if: and set no continue-on-error. Every other step in either job must be on a short setup list or be one of the two opening actions, so a step added later cannot go unpinned.

What the gating-step test does not check is documented in the test: commands are matched as substrings, so a flag or a trailing || true added to one would pass, and the setup steps are exempt by name.

I broke ci.yml in dozens of single ways (dropped steps, continue-on-error, if: false, wrong needs, a result bound to the wrong job, a narrowed aggregator if:, a name: on the aggregator, steps keyed on their first line, a step added to either job, a Windows job the aggregator does not await, and so on) and confirmed each one fails a Windows contract test. The helpers the tests rely on have their own unit tests.

Trade-offs

  • Dropping the clippy/prek wait buys a couple of minutes of wall clock on every run, paid for with Windows runner time on pushes that turn out lint-red. A middle option is needs: [changes, prek], which costs about 20 seconds; happy to switch if preferred.
  • Two windows-latest runners start instead of one, including on a docs-only pull request, where the build, test and check steps are skipped because they are gated on heavy and the jobs do little beyond setup. Fixed per-job overhead (checkout, sccache install, the cache probe, toolchain setup) is paid twice, so billed Windows time can rise while wall clock falls.
  • The rust-cache key includes the job id, so both new jobs start cold on their first runs, the old windows-build-and-test cache entry is orphaned until it ages out, and the Actions cache now holds two Windows lanes where it held one. The sccache prefix is unchanged, so the compiler cache stays warm.
  • cargo xtask e2e builds the e2e harness in the dev profile, which now happens in windows-release instead of being warmed by the debug build. sccache should absorb this after the first run.
  • The two jobs are not the same length (cold: about 14.7 minutes against about 20.5 to 21). The release build is the long step (about 11.5 minutes cold). Of the steps in the release job that need no binaries, only the pinned-key check is sizeable (about 160 s cold, about 30 s on a warm main run), and the release-readiness and installer syntax checks take seconds, so moving steps would not close the gap by itself. Re-measure on main pushes before moving anything.
  • Fork pull requests still get no shared compiler cache; that is a separate change.

Verification

  • cargo test -p xtask (353 tests), cargo fmt --check and cargo clippy -p xtask --all-targets -- -D warnings pass locally, and the leak scan and conflict-marker grep are clean.
  • The jobs and the aggregator have run on GitHub: this pull request's first run was green, including windows-debug, windows-release and windows-build-and-test. The commits added since then change only xtask test code and comments in ci.yml and the docs, and their CI run is the first check of those.
  • The workspace tests were not run locally (the local pre-push hook was skipped); CI runs the workspace.
  • actionlint was not available locally.
  • The two Strix Halo Ubuntu lanes in the separate self-hosted workflow (nightly and release) were cancelled at their 45-minute limit on this pull request. The same two lanes hit the same limit on recent main runs, and this change only edits a comment in that workflow.

This is CI plumbing and changes no CLI behaviour, so there is no Gherkin scenario.

The single windows-build-and-test job was the longest pole of every CI run
(about 16 min on main, longer on fork PRs, which get no shared compiler
cache), and it started about 3 min late because it waited for the Linux
clippy and prek jobs.

- windows-debug: clippy, dev build, workspace tests, local smoke, harness
  self-tests.
- windows-release: release build, install lifecycle e2e, pinned-key check,
  PR bundle packaging. These share no outputs with the debug side.
- windows-build-and-test stays as the required check, now an aggregator that
  needs both jobs and `changes`, and requires each to have finished exactly
  `success`. A skip is accepted only for a manual dispatch that did not ask
  for binaries. Whether the Windows jobs run is decided once, as the new
  `changes.windows` output; both jobs gate on it and the aggregator derives
  its expected skip from it, so the two cannot disagree.
- Neither Windows job waits for clippy or prek any more. Those remain
  required checks of their own, so a lint failure still blocks the merge; the
  cost is Windows runner time on pushes that turn out lint-red.

Every step of the old job is carried over unchanged into exactly one new job
(sccache stats runs in both). Contract tests in xtask pin the split: the
aggregator awaits every job in the Windows lane and its script is driven over
each possible job result; the work jobs' gating, needs and runner are pinned
exactly, with no job-level continue-on-error; the two jobs' env and setup
steps cannot drift; and each gating step is pinned by name with its command,
exact `if:` and no continue-on-error. Stale mentions of the old single job in
comments and docs are updated.

This is CI plumbing and changes no CLI behaviour, so there is no Gherkin
scenario. The new jobs have not yet run on GitHub; the first pull request run
is the only check of the live wiring.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The rewired merge-required Windows CI gate lacks a live GitHub run, so a maintainer must confirm debug/release execution, aggregator results, and manual-dispatch skip behavior.

No open issues 🎉

What changed in this PR

This PR aims to shorten CI runs by splitting Windows debug and release work into parallel jobs, while preserving windows-build-and-test as the required check.

Changes:

  • Separates debug validation from release builds, lifecycle checks, and packaging; removes both jobs’ wait for Linux lint checks.
  • Adds a shared Windows execution condition and an aggregator that checks dependency results.
  • Adds workflow contract tests and updates documentation and comments to reflect the split.
File Description
xtask/​src/​workflow_contract.rs Adds contracts for Windows job gating, aggregation, setup consistency, and retained validation steps.
xtask/​src/​env_mutation_contract.rs Updates comments identifying the job that runs Windows workspace tests.
tests/​e2e-cucumber/​features/​comfyui.feature Updates comments identifying the Windows lifecycle job.
docs/​testing.md Clarifies which Windows job runs workspace tests and lifecycle scenarios.
.github/​workflows/​e2e-selfhosted.yml Updates comments distinguishing hosted Windows jobs from GPU coverage.
.github/​workflows/​ci.yml Splits Windows work into parallel jobs and adds the required-check aggregator.

Review effort: Balanced

@jussielo-amd
jussielo-amd marked this pull request as ready for review October 30, 2026 12:43
@jussielo-amd
jussielo-amd requested a review from a team as a code owner October 30, 2026 12:43

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR and found nothing blocking. Two small notes are left inline.

What I checked:

  • The aggregator job windows-build-and-test keeps its name, so the required-check match still holds. It needs changes, windows-debug and windows-release, uses !cancelled(), and requires an exact result from each.
  • Every old step now lives in exactly one of the new jobs, and the release job keeps its AWS_LC_SYS_USE_SYSTEM env.
  • Action pins are unchanged, no stale references to the old single job remain in docs, features or xtask, and I saw no leaks.

Notes:

  • On this PR's cold-cache run windows-release took 20m31s against 14m46s for windows-debug, so the lanes are unbalanced and the gain is still unmeasured.
  • The two new work jobs are not required checks themselves, so the aggregator needs to stay in the branch-protection ruleset.
  • Dropping the wait on clippy/prek is a documented, deliberate tradeoff.

Not verified: the "41 mutation checks" claim, and I did not run the xtask tests. Two Strix Halo Ubuntu lanes (nightly, release) failed at 45m25s on files this PR does not touch, likely runner infra, but I did not confirm the cause. This reflects a diff read, not a full CI result.

Comment thread .github/workflows/ci.yml Outdated
# Release side of the Windows lane. It builds the optimised binaries, and the
# install lifecycle and the PR bundle consume them. The release-readiness
# self-test, the installer syntax check and the pinned-key check need no binaries;
# they sit here only to keep the two jobs about the same length. See

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On this PR's run windows-release took 20m31s vs windows-debug 14m46s, so the two are not about the same length yet (cold caches on a PR). Worth re-measuring on a main push and rebalancing if the release side stays the long pole.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that the lanes are not balanced yet, and that one cold run is not enough to rebalance on. Measured so far on cold pull-request runs: windows-debug about 14.7 min and windows-release about 20.5 to 21 min, and the run reached its last non-GPU job after about 22 min. The release build is the long step (about 11.5 min cold), and of the steps in the release job that need no binaries only the pinned-key check is sizeable (about 160 s cold, about 30 s on warm main runs), so moving steps would not close the gap on its own. I have not moved anything and have left the comment above windows-debug saying the warm-cache balance is unmeasured; the plan is to compare several push runs on main after merge (per-job durations and time until the last non-GPU job) before rebalancing. The comment change is in 0a04bcb.

Comment thread docs/testing.md Outdated

The required `windows-build-and-test` lane additionally folds in the hermetic
The `windows-release` job of the required `windows-build-and-test` lane additionally
folds in the hermetic

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the rewrap leaves "folds in the hermetic" as an orphan line; reflow the paragraph.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reflowed in ab28aad. The same short orphan line was in four places, so I reflowed all of them (two paragraphs in docs/testing.md, the comment in comfyui.feature, and the module comment in env_mutation_contract.rs) with no wording change.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · ce854ce

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (base df1548a2…head ce854ce9). Purely CI plumbing: splits the Windows GitHub Actions lane into two parallel jobs behind a pinned aggregator; no ticket named in the description, and none appears needed for a CI-only change.

Blocking

None

Non-blocking

  • ci_jobs()'s comment-attachment differs from the pre-existing job_block()'s, which is a latent fragility in the new contract-test helpers — xtask/src/workflow_contract.rs:4252-4277 (new ci_jobs/ci_job) vs xtask/src/workflow_contract.rs:303-320 (pre-existing job_block).
    job_block stops a job's extracted text at the first indent-2 line, comment or not, so a trailing comment before the next job is excluded from both jobs. ci_jobs only starts a new job at an indent-2, non-comment line, so a trailing comment block that precedes a job header (e.g. the rationale comment right before windows-build-and-test:, which itself contains the literal strings !cancelled() and always()) is folded into the previous job's text instead of being excluded or attached to the job it actually describes. ci_windows_aggregator_requires_every_windows_job then does code.contains("!cancelled()") && !code.contains("always()") over ci_job(&ci, "windows-build-and-test"); today this still evaluates correctly only because the job's own if: line repeats !cancelled() and the misattributed comment's always() mention lands in windows-release's captured block instead, outside what the assertion checks. A later edit to that comment, or a new trailing comment before any job this helper is also used on, could make the assertion pass or fail for the wrong reason without anyone noticing, which runs against the rest of this test file's evident care about not trusting a check for the wrong reason.
    Confidence 70 · architectural · Fix: make ci_jobs's job-boundary rule match job_block's (stop at any indent-2 line), or have it drop a job's leading comment lines instead of attaching them to whichever job happened to be parsed immediately before.

Decisions for the author

  • Dropping windows-debug/windows-release's needs: [changes, clippy, prek] relies on clippy and prek already being independently-required branch-protection checks — tradeoff. ci.yml's own comment states "Those stay required checks in their own right, so a lint failure still blocks the merge," which is a claim about branch-protection configuration that isn't visible from the files in this repo. It is very likely already true independent of this change — dozens of other jobs in the same file (e.g. test, build-and-test) still need: [changes, clippy, prek] purely as a compute-saving gate, a pattern that only makes sense if clippy/prek are already required on their own — and the pre-split job's if: treated a clippy/prek skip the same as success, so this PR does not newly introduce reliance on "skipped reads as passing" for the Windows lane; if anything it now produces a real success/failure for windows-debug/windows-release instead of skipping them outright when lint is red. Worth a quick confirmation against the actual ruleset before merge, since nothing in-repo proves it.
  • Running two parallel windows-latest jobs instead of one serial job duplicates per-job fixed overhead (checkout, sccache install, the cache-backend probe, toolchain setup) that was previously paid once — tradeoff. The change's own comments already weigh the clippy/prek-wait removal's cost in wall-clock terms, but not this one in Windows-runner-minutes: total billed Windows compute time may rise somewhat even as end-to-end latency falls. Worth confirming that's an accepted cost, alongside the already-documented cold-cache effect on both jobs' first runs.

Positive signals

  • ci_windows_aggregator_passes_only_when_every_job_succeeded directly executes the aggregator's extracted bash script under bash -e -c against every result/skip combination (success/failure/cancelled/skipped × expected-skip true/false) rather than only asserting on source text — real behavioral verification of the merge gate, independently re-run and confirmed correct for this review.
  • Every step from the pre-split monolithic job lands in exactly one of windows-debug/windows-release, confirmed by diffing the old single job against the new pair step-by-step; nothing was silently dropped or duplicated.
  • The four supporting files (docs/testing.md, tests/e2e-cucumber/features/comfyui.feature, xtask/src/env_mutation_contract.rs, .github/workflows/e2e-selfhosted.yml) were each updated to name the correct new job for the behavior they describe — including the non-obvious case that the install-lifecycle/E2E_ALSO_RUN_IDS work moved to windows-release while the single-process cargo test --workspace moved to windows-debug, which would have been an easy pair to swap by mistake.
  • ci_windows_jobs_share_one_cache_and_toolchain_setup guards the two duplicated jobs' env and setup steps against silent drift from each other, and ci_windows_jobs_keep_every_gating_step_enabled pins every gating step's exact command, if:, and absence of continue-on-error — both checked by direct reading against the literal YAML, including the trickier >- block-scalar steps.
  • All uses: lines are pinned to full commit SHAs with trailing version comments, consistent with this repo's own convention; single, clearly-written, DCO-signed commit whose message matches the diff.

Deployment notes

  • At head commit ce854ce9, two self-hosted GPU jobs in the separate "E2E self-hosted" workflow — "E2E tests (Strix Halo, Ubuntu, nightly, current)" and "...release, current)" — show cancelled, and two more ("E2E tests (MI300X, current)", "E2E tests (W7900, current)") remain queued. This PR touches that workflow with only a verified-accurate comment edit, the job logs show an artifact upload immediately followed by orphan-process termination (consistent with a runner-side cancellation rather than a failing assertion), and this repo's own AGENTS.md documents that self-hosted GPU hardware availability can differ from a clean run. All checks this change can actually affect — the windows-build-and-test aggregator, windows-debug, windows-release, clippy, prek, and the affected-crates test job that compiles/runs the new contract tests — are green. Recommend confirming the self-hosted lanes are unrelated infrastructure flakiness before treating the run as fully clean.

What this covered

Every one of the 6 changed files read completely at the PR's head content (fetched via git show refs/pr810head:<path>, never checked out), plus unchanged repository context (module wiring in xtask/src/main.rs, release.yml/nightly.yml, README.md, other docs) for impact analysis — via two independent blind fan-out workers (one on ci.yml + xtask/src/workflow_contract.rs, one on the four smaller doc/comment files), each scoped to files only, with no access to this PR's description or comments, plus my own consolidation, impact-analysis and design-review pass. Range reviewed: df1548a211ed5a53d37d2e5df8f83a7746a13239…ce854ce901021150f06ba765d8286e0e197a5e9e.

Procedural note: as the orchestrating context I necessarily read this PR's title/body once during the mandatory pre-review duplicate-review check (the API call that returns the head SHA also returns the body); neither fan-out worker saw it, and its content was not used for judgment until after the findings above were settled.

Live CI was read at the exact head commit (gh pr checks): all Windows- and lint-relevant jobs are SUCCESS, as detailed under Deployment notes. "Prior changes" — no earlier review threads exist on this PR to reconcile against (confirmed zero review-comment threads). "History" — compared the pre-split job definition on origin/main against the new pair directly via git show/git diff. Could not execute cargo test -p xtask directly (no Rust toolchain available in this environment); relied on the live CI's own passing "Test (affected crates)" job plus direct re-execution of the aggregator's bash script and manual tracing of the YAML-parsing helpers against the literal file content. Not reconciled against any discussion on this PR.

…vious job

The Windows-split rewording left short orphan lines in three prose
paragraphs; reflow them with no wording change.

ci_jobs() left the comment block above a job on the end of the job before
it, so that block carried text that describes its neighbour. Drop the
comment and blank lines between jobs, as job_block already does, and pin
it with a test that also covers a nested comment on a job's last line.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…step

The aggregator tests only looked for `needs.<job>.result` somewhere in the
job, so binding the release result to the wrong variable, narrowing the
job's `if:`, or adding `if`/`continue-on-error`/`shell` to its step all kept
the required check green while letting a failed Windows job through.

Assert that each Windows job's result is bound to exactly one variable the
script checks, that the job's `if:` is exactly `!cancelled()`, and that it
is one plain step on a Linux runner. The script test now takes its variable
names from those bindings. The per-job checks cover every job found by
runner instead of a fixed pair, with a fixture test for that lookup.

Also say in the ci.yml comment that the release side now compiles the e2e
harness's dev-profile dependencies itself.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…stale claims

The aggregator and gating tests looked for `if` and `continue-on-error` only
where a step's keys sit after its first line, so a step written as
`- if: false` followed by `name:` passed unseen. Compare the exact key list
of the aggregator's job and step instead, which also pins that the job has
no `name:` (it would rename the required check), and read gating steps' keys
with step_keys. Read the aggregator's env and script through the existing
step_env and preflight_body_code instead of new readers.

Every step in both Windows jobs must now be in the gating table or on a short
list of setup steps, so a step added later cannot go unpinned.

Reword four ci.yml comments the split made wrong: the Windows build time in
the timeout note, the `windows` output not being a path filter, the heavy-job
gating note on the remote-control job, and why the pinned-key check sits in
the release job.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…helpers

The gating-step test's doc claimed more than it checks: commands are matched
by substring and the setup steps are exempt by name, so a flag or a trailing
`|| true` passes. Say so, rather than leave a claim the test cannot fail on.

Pin the helpers the aggregator tests depend on, so weakening one cannot hide
a break in ci.yml: a command that only a comment mentions, a second step in
the aggregator, and an env expression bound twice or not at all.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…per tests

The step-completeness check accepted any unnamed step whose raw text mentioned
one of the two opening actions, comments included, with any keys. Match the
`uses:` value, allow only `uses`, `with` and an absent-or-`heavy` `if`, and say
in the test's doc what is and is not checked there.

The helper tests now also cover a value that only contains the expected
expression, and `code_only` keeping one output line per input line. Correct
the comment naming which tests use which helper.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…toolchain step by `uses:`

The completeness check accepts any number of unnamed checkout or
setup-rust-toolchain steps at any position, not only the two that open each
job; say so in the doc and inline comments rather than claim more.

Find the toolchain step in the shared-setup test by its `uses:` value instead
of by substring, so a comment naming the action in an earlier step cannot end
the compared region early. Say in code_only's doc that it keeps one output
line per input line, which the helper test pins.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this against the saved diff and found nothing that blocks the merge. I'm requesting changes on the points below, which I think are worth settling first.

Wall-clock gain isn't shown yet. On this PR's own cold-cache run windows-release took 20m31s against 14m46s for windows-debug, so the long pole just moved and it's no better than the old 20+ minute single job for fork PRs. The "about 10 min on main" figure is an estimate for a warm cache, not something measured. Either rebalance the lanes (e.g. move the release-readiness self-test or the harness self-tests across) or re-measure on main before claiming it.

Contract-test cost. About 740 added lines of hand-rolled YAML scraping (indent-based yaml_value_at/ci_jobs, strip_comment) for a two-job split. I understand the required-check risk justifies some of it, but it's brittle against indentation changes and will be a maintenance tax.

Gating test is loose. It matches commands by substring, so a trailing || true on a gating step would still pass. You've disclosed this; I'd tighten it if it's cheap, otherwise leave a note next to the test.

Comment drift. The timeout note still says "roughly 4x the observed p100" while listing Windows at 15-20 min with a 45 min limit (see inline).

Decisions that are yours to make (all disclosed in the PR body, so not asking for changes): dropping the wait on clippy/prek, paying for two Windows runners even on docs-only PRs, and starting both jobs on a cold per-job rust-cache key.

Not checked: I only read the diff, so I didn't grep the rest of the repo (docs, scripts, other workflows) for stale references to the old single Windows job, and I didn't re-run the "broke ci.yml dozens of ways" mutation checks. Worth a grep on your side. CI was still running at this head (the Windows jobs and the aggregator hadn't reported), so this isn't a read on those results.

Comment thread .github/workflows/ci.yml Outdated
@@ -71,17 +71,21 @@ env:
# with no timeout — holds a runner for six hours and, in the merge queue, stalls
# every queued PR behind it with no signal that anything is wrong. The values here
# are deliberately loose (roughly 4x the observed p100, which is under 3 min for

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This note says the limits are "roughly 4x the observed p100", but the Windows entry below it is 15-20 min against a 45 min limit (about 2x). Could you reword so the comment doesn't contradict the value?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair, the wording contradicted the value. Reworded in 0a04bcb: the roughly 4x figure is for most jobs, and the two Windows jobs are called out as the exception at about 2 to 3x (15 to 21 min against 45 min on a cold pull-request cache).

Comment thread .github/workflows/ci.yml Outdated
# of the whole run (it alone decided when CI went green); the debug side
# (clippy, build, test, smoke) and the release side (release build, install
# lifecycle, PR bundle) share no outputs, so they run side by side. The one
# thing the release side gives up is the debug build's warm-up of the e2e

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This describes the dev-profile warm-up as the only thing the release lane gives up, but on this PR's cold run the release lane took 20m31s versus 14m46s for debug. Consider rebalancing (e.g. move the release-readiness self-test or the harness self-tests) so the two lanes are closer, or note that the gain depends on a warm cache.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right that this overstated it. Reworded in 0a04bcb: the comment now says the release side also gives up the debug build's warm-up, records that on a cold pull-request cache release took about 21 min against about 15 for debug, and says the warm-cache balance has not been measured since the split. I did not rebalance because the movable steps are small (see my reply on the earlier thread); the plan is to compare several push runs on main first. I also dropped a claim in the pinned-key comment that it reuses the release build, since that step took minutes on the cold run and I cannot show from the log that it reuses anything.

The timeout note called every limit roughly 4x the observed p100 while
listing a 15-20 min Windows job against a 45 min limit. Say the Windows jobs
are the exception.

The split comment said the release side gives up only the dev-profile
warm-up, but on a cold pull-request cache the release job took about 21 min
against about 15 for debug. State that, and that the warm-cache balance is
unmeasured. The pinned-key comment claimed it reuses the release build; on a
cold cache the step took minutes, so say only what is known.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · e311e0e

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (base df1548a2…head e311e0e2, 6 files, 977/70 +/-). Purely CI plumbing: splits the Windows GitHub Actions lane into two parallel jobs (windows-debug, windows-release) behind a pinned windows-build-and-test aggregator that keeps the required-check name. No ticket is named in the description, and none appears needed — AGENTS.md §3 exempts "purely internal changes (refactors, CI plumbing, docs)" from a Gherkin scenario provided the PR text says why, which it does.

Blocking

None

Non-blocking

None

Decisions for the author

  • The claim that clippy/prek are already independently-required branch-protection checks is unverifiable from the repo itself — tradeoff. ci.yml's own comment on windows-debug states they "stay required checks in their own right, so a lint failure still blocks the merge." I traced GitHub Actions' actual if: semantics (a custom if: that doesn't use success()/failure()/cancelled()/always() is still implicitly ANDed with success() of every job in needs:) and confirmed windows-debug/windows-release's if: needs.changes.outputs.windows == 'true' does still require changes to succeed, so the mechanism is sound — but whether clippy and prek are themselves required checks in the branch-protection ruleset (so that dropping them from needs: doesn't let a lint failure merge) is a GitHub setting, not something any file here proves. Worth a quick confirmation against the actual ruleset before merge.
  • Lane rebalancing is an open, self-acknowledged follow-up — tradeoff, not a defect. The PR's own cold-cache run shows windows-release (20m31s) now the longer pole, versus windows-debug (14m46s); the "~10 min on main" estimate is explicitly flagged in the PR body as an unmeasured, warm-cache projection. Nothing about the split logic is wrong — gating, aggregation, and step placement all check out (see below) — this is purely a question of whether the wall-clock goal is met yet, which only a warm main run (or deliberately moving a self-test across jobs) can answer.

Positive signals

  • The ci_jobs/ci_job YAML-block-extraction helpers (xtask/src/workflow_contract.rs:4254-4286) correctly drop the comment-and-blank-line region between two job headers rather than attaching it to either neighbor, with a dedicated regression test (ci_jobs_drops_the_comment_between_jobs_instead_of_keeping_it_on_the_job_before) covering a nested end-of-job comment and a trailing-comment job header containing the literal strings !cancelled()/always(). I traced the loop by hand against that fixture and confirmed the boundary logic is sound.
  • ci_windows_aggregator_passes_only_when_every_job_succeeded extracts the aggregator's actual bash script from the live YAML and executes it under real bash -e -c across every (changes, windows-debug, windows-release, expect-skip) combination that matters, rather than only pattern-matching source text — I re-derived the combinatorics by hand and the script's pass/fail verdict is correct in every case I checked, including a failed changes job under an expected-skip scenario.
  • Every GATING-table entry in ci_windows_jobs_keep_every_gating_step_enabled matches an actual step in the current windows-debug/windows-release blocks, and the test's own doc comment accurately discloses its real limitation (substring command matching, name-exempt setup steps) rather than overclaiming — I checked the disclosed gap against the implementation and it is not understated.
  • The four small supporting files (docs/testing.md, comfyui.feature, env_mutation_contract.rs, e2e-selfhosted.yml) each attribute the right behavior to the right new job (e.g. E2E_ALSO_RUN_IDS to windows-release, the single-process cargo test --workspace to windows-debug) and a full-file grep turned up no stale leftover mention of the old single job.
  • Live CI at this exact head is green on every check the change can affect: windows-build-and-test, Windows build and test (debug), Windows release build and lifecycle, clippy, prek, Test (affected crates), build-and-test, Commit signatures + sign-off.

Deployment notes

At head e311e0e2, the separate self-hosted "E2E tests" workflow (which this PR touches only via a verified-accurate comment edit) has two GPU lanes still queued/in-progress (MI300X, W7900, and the nightly Strix-Halo-Ubuntu lane); these are optional/non-blocking per that workflow's own code comment ("Non-blocking so it never gates the PR") and unaffected by this change. No action needed before merge on that basis.

What this covered

DEGRADED REVIEW — three parallel fan-out spawns (Agent tool) all failed on a shared, cross-session subagent concurrency limit ("Concurrent subagent limit reached... Do not retry"); ListAgents confirmed 20+ agents running under other sessions on this machine, none mine. Per the review method, this was not retried, and the review proceeded single-context instead, with real correctness tracing (not just reading) to compensate: GitHub Actions' if:/implicit-success() semantics were independently verified against current GitHub docs rather than assumed, the ci_jobs comment-boundary fix was traced by hand against its fixture, and the aggregator's bash script was re-derived combination-by-combination by reasoning (not re-executed, since no local Rust/bash toolchain is available in this environment — the live CI run's own green "Test (affected crates)" job is relied on for actual execution). Recommend re-running with fan-out once subagent capacity is available, for broader coverage than one context can provide.

Read completely: .github/workflows/ci.yml (full file, ~1850 lines, plus its diff), xtask/src/workflow_contract.rs's entire changed region (the Windows-lane tests and the helpers they depend on, old and new) plus the pre-existing helper functions (job_block, strip_comment, indent_of, run_block, step_env, step_keys) those tests reuse, and all four small supporting files in full. Commit-by-commit history of the six follow-up commits since the prior reviewed head was read via their messages (not full git blame). Checked live CI (get_check_runs) and AGENTS.md for this repo's own conventions.

Did not run: "Prior changes" — no applicable prior review thread was reconciled against (per the review method, discussion is read only after findings are settled, and is not referenced here). Could not execute cargo test -p xtask, cargo clippy, or the aggregator's bash script directly (no local toolchain in this sandboxed environment); relied on live CI's passing "Test (affected crates)" job plus manual tracing instead. Not reconciled against any discussion on this PR.


🤖 Generated with Claude Code

@jussielo-amd

Copy link
Copy Markdown
Collaborator Author

Thanks for the review. What changed since it, and where I left things:

Wall-clock gain. The body now states what has been measured (cold pull-request runs: debug about 14.7 min, release about 20.5 to 21 min, last non-GPU job after about 22 min) and says the gain on main has not been measured since the split. I would rather re-measure over several push runs on main after merge than move steps on one cold run; details in the reply on the inline thread.

Comment drift. Both inline comments are fixed in 0a04bcb. I also grepped docs/, README.md, .github/, xtask/ and tests/ for the old single-job wording and the old claims; the stale ones were four comments in ci.yml, which are reworded, and .pre-commit-config.yaml still says the workspace tests run in build-and-test, which was already inaccurate before this change and I have left alone.

Gating test is loose. Agreed. It is documented in the test's doc comment (substring match on the command, the setup steps exempt by name), so a trailing || true on a gating step would pass. Matching the run block exactly would make the test fail on any harmless command edit, so I left it loose and noted it rather than tighten it here.

Contract-test cost. Fair point. The helpers added since your read reuse the existing step_env, step_keys and preflight_body_code rather than new scrapers, but ci_jobs is still a second job-block extractor next to job_block, and I have not consolidated them in this change.

Mutation checks. I re-ran them against the current head: each single breakage I tried (a result bound to the wrong job, a narrowed aggregator if:, a name: on the aggregator, if/continue-on-error/shell on the aggregator step or written on a step's first line, an extra step in either Windows job) fails a Windows contract test.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · 0a04bcb

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (base df1548a2…head 0a04bcb7, 6 files, +984/-71, 8 commits). Purely CI plumbing: splits the Windows GitHub Actions lane into two parallel jobs (windows-debug, windows-release) behind a pinned windows-build-and-test aggregator that keeps the required-check name. No ticket named; the PR body states this is CI plumbing changing no CLI behaviour, so no Gherkin scenario is needed (AGENTS.md §3's exemption), and the description's stated scope (984/+71 across 6 files, 8 commits) matches the diff exactly.

Blocking

None

Non-blocking

None

Decisions for the author

  • The claim that clippy/prek are already independently-required branch-protection checks is unverifiable from the repo itself — tradeoff. ci.yml's comment on windows-debug/windows-release states they "stay required checks in their own right, so a lint failure still blocks the merge." The if: mechanism is sound (needs.changes.outputs.windows == 'true' still implicitly requires changes to succeed), but whether clippy/prek are themselves required in the branch-protection ruleset is a GitHub setting no file here proves. Worth a quick confirmation against the actual ruleset before merge — this has been raised at each prior head reviewed and still has no in-repo answer, though it is unchanged risk rather than a new one.

Positive signals

  • The PR body's "Trade-offs" section discloses exactly what the diff's own ci.yml comments disclose — duplicated per-job fixed overhead (checkout/sccache/toolchain paid twice), the rust-cache key change, the lane-length imbalance (14.7 vs 20.5-21 min cold), and the clippy/prek-wait removal — with no gap between stated and actual tradeoffs.
  • Every step from the pre-split monolithic Windows job lands in exactly one of windows-debug/windows-release; confirmed by direct diff of the old single job against the new pair.
  • The aggregator's bash script (windows-build-and-test's one step) was traced and executed by hand against the literal YAML across every success/failure/cancelled/skipped × expected-skip combination; the pass/fail verdict is correct in each case, including a failed changes job under an expected-skip scenario.
  • The four small supporting files (docs/testing.md, comfyui.feature, env_mutation_contract.rs, e2e-selfhosted.yml) each attribute the right post-split behavior to the right job name (e.g. E2E_ALSO_RUN_IDS/install-lifecycle to windows-release, cargo test --workspace to windows-debug, an easy pair to get backwards), verified against the actual job bodies; a repo-wide grep found no other stale reference to the old single Windows job.
  • Commit history is 8 small, specifically-worded, DCO-signed commits that progressively tighten the new contract tests (pinning aggregator keys, step keys, gating-step coverage) rather than noise; the final 2 commits touch only comments, confirmed by diffing each commit against ci.yml.

Deployment notes

Live CI at this exact head is green on all 25 reported checks, including windows-build-and-test, Windows build and test (debug), Windows release build and lifecycle, clippy, prek, Test (affected crates), and Commit signatures + sign-off. No pending or cancelled lanes at this head (unlike earlier heads of this PR, where self-hosted GPU lanes were still queued/cancelled).

What this covered

Every one of the 6 changed files read completely at this head, via two independent blind fan-out workers (one on ci.yml + xtask/src/workflow_contract.rs as a coupled group, one on the four smaller supporting files), each with no access to this PR's description or comments, plus my own direct reading of the Windows job definitions and aggregator logic in ci.yml, a commit-by-commit diff of all 8 commits (confirming the 7 follow-up commits after the initial split touch only xtask tests and comments, never production YAML logic), and a repo-wide grep for stale references to the old single Windows job (none found, aside from an unrelated identically-named windows-release job in the separate release.yml release-publishing workflow). Range reviewed: df1548a211ed5a53d37d2e5df8f83a7746a13239…0a04bcb7e03af879ad4e1602e2f89adc7b49137d.

Live CI was read via get_check_runs at the exact head commit. Could not execute cargo test -p xtask or cargo clippy directly (no Rust toolchain in this sandboxed environment); relied on live CI's own green "Test (affected crates)"/"clippy" jobs plus direct hand-tracing/execution of the aggregator's extracted bash script against the literal YAML instead. "Prior changes" — the same two architectural/tradeoff themes (clippy/prek reliance, lane imbalance) have been raised at each previously-reviewed head of this PR by this same automation and by a human reviewer; both remain author decisions rather than defects, and the lane-imbalance one is now explicitly re-disclosed in this head's own updated comments. Not reconciled against any discussion on this PR.


🤖 Generated with Claude Code

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 0a04bcb. No blocking findings.

The required windows-build-and-test context is unchanged: the aggregator keeps the job id with no name:, runs under !cancelled(), compares each result by exact equality, and derives its expected skip from the same changes.windows output the work jobs gate on. A failed or skipped changes fails it. Pinned action SHAs are preserved, and all checks are green.

Non-blocking:

  • The measured gain is only shown for warm main; on a cold fork-PR cache the release lane alone (~20-21 min) is about the length of the old single job, so the split mainly saves the clippy/prek wait.
  • Measured timings hardcoded in ci.yml comments will go stale.
  • Only docs/testing.md is updated; worth a repo-wide grep for other docs that still describe windows-build-and-test as a single job.

For you to decide: dropping the clippy/prek wait costs two Windows runners on lint-red pushes (needs: [changes, prek] is the cheap alternative), and the ~90 lines of duplicated setup across the two jobs is held together by the drift test.

Comment thread .github/workflows/ci.yml Outdated
# lifecycle, PR bundle) share no outputs, so they run side by side. The release
# side gives up the debug build's warm-up of the e2e harness's dev-profile
# dependencies, which it now compiles for itself. The two sides are not the same
# length: on a cold pull-request cache the release job took about 21 min against

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: these measured durations (about 21 vs 15 min) will go stale in a comment; consider dropping the numbers and pointing at this PR. Also, for cold-cache fork PRs the release lane alone is about the length of the old single job, so the split only helps there via the dropped clippy/prek wait.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped the minutes from both comments in 95baf4d (this one and the timeout note, which had the same problem); they now say the release job is the longer one on a cold cache and point at the numbers on this pull request. On the cold fork-PR point, the two pre-split cold PR runs I checked had the single Windows job at 1593 s and 1754 s, after waiting about 340 s for the lint jobs. On this pull request's cold runs the release job took 1231 s and 1257 s and started after about 50 s, so the release lane alone is shorter than the old job, and the whole run reached its last non-GPU job about 10 to 13 minutes sooner (about 1320 s against 1950 to 2090 s). It is two data points per side, so I would still confirm on warm main runs after merge.

Comment thread .github/workflows/ci.yml
# been measured since the split; compare the two jobs' durations over several
# runs there before rebalancing steps between them.
#
# Neither job waits for `clippy` or `prek`. Those stay required checks in their

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tradeoff to confirm: neither Windows job waits on clippy/prek any more, so a prek failure (rustfmt, ruff, shellcheck) costs two Windows runners on lint-red pushes. The body offers needs: [changes, prek] (~20 s); worth deciding explicitly.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping it as it is, since the body discloses the cost and you list it as a decision rather than a defect. For the record, clippy and prek (lint / hygiene) are required checks on this pull request (I read isRequired from GraphQL for the previous head), so a lint failure still blocks the merge. The cost is the two Windows runners on lint-red pushes, and needs: [changes, prek] stays available if you would rather pay about 20 s of wall clock to avoid it.

Minutes quoted in a comment go stale as the runners and caches change. Say
which Windows job is the longer one and where to compare the two, without
the numbers, in both the timeout note and the split note.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@jussielo-amd
jussielo-amd added this pull request to the merge queue Nov 2, 2026
Merged via the queue into ROCm:main with commit 5a0742f Nov 2, 2026
26 checks passed
@jussielo-amd
jussielo-amd deleted the ci/windows-parallel-jobs branch November 2, 2026 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants