Repository navigation
ci(windows): run Windows debug and release builds as parallel jobs - #810
Conversation
The single windows-build-and-test job was the longest pole of every CI run (about 16 min on main, longer on fork PRs, which get no shared compiler cache), and it started about 3 min late because it waited for the Linux clippy and prek jobs. - windows-debug: clippy, dev build, workspace tests, local smoke, harness self-tests. - windows-release: release build, install lifecycle e2e, pinned-key check, PR bundle packaging. These share no outputs with the debug side. - windows-build-and-test stays as the required check, now an aggregator that needs both jobs and `changes`, and requires each to have finished exactly `success`. A skip is accepted only for a manual dispatch that did not ask for binaries. Whether the Windows jobs run is decided once, as the new `changes.windows` output; both jobs gate on it and the aggregator derives its expected skip from it, so the two cannot disagree. - Neither Windows job waits for clippy or prek any more. Those remain required checks of their own, so a lint failure still blocks the merge; the cost is Windows runner time on pushes that turn out lint-red. Every step of the old job is carried over unchanged into exactly one new job (sccache stats runs in both). Contract tests in xtask pin the split: the aggregator awaits every job in the Windows lane and its script is driven over each possible job result; the work jobs' gating, needs and runner are pinned exactly, with no job-level continue-on-error; the two jobs' env and setup steps cannot drift; and each gating step is pinned by name with its command, exact `if:` and no continue-on-error. Stale mentions of the old single job in comments and docs are updated. This is CI plumbing and changes no CLI behaviour, so there is no Gherkin scenario. The new jobs have not yet run on GitHub; the first pull request run is the only check of the live wiring. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🔵 Needs a closer look
The rewired merge-required Windows CI gate lacks a live GitHub run, so a maintainer must confirm debug/release execution, aggregator results, and manual-dispatch skip behavior.
No open issues 🎉
What changed in this PR
This PR aims to shorten CI runs by splitting Windows debug and release work into parallel jobs, while preserving windows-build-and-test as the required check.
Changes:
- Separates debug validation from release builds, lifecycle checks, and packaging; removes both jobs’ wait for Linux lint checks.
- Adds a shared Windows execution condition and an aggregator that checks dependency results.
- Adds workflow contract tests and updates documentation and comments to reflect the split.
| File | Description |
|---|---|
xtask/src/workflow_contract.rs |
Adds contracts for Windows job gating, aggregation, setup consistency, and retained validation steps. |
xtask/src/env_mutation_contract.rs |
Updates comments identifying the job that runs Windows workspace tests. |
tests/e2e-cucumber/features/comfyui.feature |
Updates comments identifying the Windows lifecycle job. |
docs/testing.md |
Clarifies which Windows job runs workspace tests and lifecycle scenarios. |
.github/workflows/e2e-selfhosted.yml |
Updates comments distinguishing hosted Windows jobs from GPU coverage. |
.github/workflows/ci.yml |
Splits Windows work into parallel jobs and adds the required-check aggregator. |
Review effort: Balanced
juhovainio
left a comment
There was a problem hiding this comment.
I reviewed this PR and found nothing blocking. Two small notes are left inline.
What I checked:
- The aggregator job
windows-build-and-testkeeps its name, so the required-check match still holds. It needschanges,windows-debugandwindows-release, uses!cancelled(), and requires an exact result from each. - Every old step now lives in exactly one of the new jobs, and the release job keeps its
AWS_LC_SYS_USE_SYSTEMenv. - Action pins are unchanged, no stale references to the old single job remain in docs, features or xtask, and I saw no leaks.
Notes:
- On this PR's cold-cache run
windows-releasetook 20m31s against 14m46s forwindows-debug, so the lanes are unbalanced and the gain is still unmeasured. - The two new work jobs are not required checks themselves, so the aggregator needs to stay in the branch-protection ruleset.
- Dropping the wait on clippy/prek is a documented, deliberate tradeoff.
Not verified: the "41 mutation checks" claim, and I did not run the xtask tests. Two Strix Halo Ubuntu lanes (nightly, release) failed at 45m25s on files this PR does not touch, likely runner infra, but I did not confirm the cause. This reflects a diff read, not a full CI result.
| # Release side of the Windows lane. It builds the optimised binaries, and the | ||
| # install lifecycle and the PR bundle consume them. The release-readiness | ||
| # self-test, the installer syntax check and the pinned-key check need no binaries; | ||
| # they sit here only to keep the two jobs about the same length. See |
There was a problem hiding this comment.
On this PR's run windows-release took 20m31s vs windows-debug 14m46s, so the two are not about the same length yet (cold caches on a PR). Worth re-measuring on a main push and rebalancing if the release side stays the long pole.
There was a problem hiding this comment.
Agreed that the lanes are not balanced yet, and that one cold run is not enough to rebalance on. Measured so far on cold pull-request runs: windows-debug about 14.7 min and windows-release about 20.5 to 21 min, and the run reached its last non-GPU job after about 22 min. The release build is the long step (about 11.5 min cold), and of the steps in the release job that need no binaries only the pinned-key check is sizeable (about 160 s cold, about 30 s on warm main runs), so moving steps would not close the gap on its own. I have not moved anything and have left the comment above windows-debug saying the warm-cache balance is unmeasured; the plan is to compare several push runs on main after merge (per-job durations and time until the last non-GPU job) before rebalancing. The comment change is in 0a04bcb.
|
|
||
| The required `windows-build-and-test` lane additionally folds in the hermetic | ||
| The `windows-release` job of the required `windows-build-and-test` lane additionally | ||
| folds in the hermetic |
There was a problem hiding this comment.
Nit: the rewrap leaves "folds in the hermetic" as an orphan line; reflow the paragraph.
There was a problem hiding this comment.
Reflowed in ab28aad. The same short orphan line was in four places, so I reflowed all of them (two paragraphs in docs/testing.md, the comment in comfyui.feature, and the module comment in env_mutation_contract.rs) with no wording change.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · ce854ce
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change (base df1548a2…head ce854ce9). Purely CI plumbing: splits the Windows GitHub Actions lane into two parallel jobs behind a pinned aggregator; no ticket named in the description, and none appears needed for a CI-only change.
Blocking
None
Non-blocking
ci_jobs()'s comment-attachment differs from the pre-existingjob_block()'s, which is a latent fragility in the new contract-test helpers —xtask/src/workflow_contract.rs:4252-4277(newci_jobs/ci_job) vsxtask/src/workflow_contract.rs:303-320(pre-existingjob_block).
job_blockstops a job's extracted text at the first indent-2 line, comment or not, so a trailing comment before the next job is excluded from both jobs.ci_jobsonly starts a new job at an indent-2, non-comment line, so a trailing comment block that precedes a job header (e.g. the rationale comment right beforewindows-build-and-test:, which itself contains the literal strings!cancelled()andalways()) is folded into the previous job's text instead of being excluded or attached to the job it actually describes.ci_windows_aggregator_requires_every_windows_jobthen doescode.contains("!cancelled()") && !code.contains("always()")overci_job(&ci, "windows-build-and-test"); today this still evaluates correctly only because the job's ownif:line repeats!cancelled()and the misattributed comment'salways()mention lands inwindows-release's captured block instead, outside what the assertion checks. A later edit to that comment, or a new trailing comment before any job this helper is also used on, could make the assertion pass or fail for the wrong reason without anyone noticing, which runs against the rest of this test file's evident care about not trusting a check for the wrong reason.
Confidence 70 · architectural · Fix: makeci_jobs's job-boundary rule matchjob_block's (stop at any indent-2 line), or have it drop a job's leading comment lines instead of attaching them to whichever job happened to be parsed immediately before.
Decisions for the author
- Dropping
windows-debug/windows-release'sneeds: [changes, clippy, prek]relies onclippyandprekalready being independently-required branch-protection checks — tradeoff.ci.yml's own comment states "Those stay required checks in their own right, so a lint failure still blocks the merge," which is a claim about branch-protection configuration that isn't visible from the files in this repo. It is very likely already true independent of this change — dozens of other jobs in the same file (e.g.test,build-and-test) stillneed: [changes, clippy, prek]purely as a compute-saving gate, a pattern that only makes sense ifclippy/prekare already required on their own — and the pre-split job'sif:treated aclippy/prekskip the same as success, so this PR does not newly introduce reliance on "skipped reads as passing" for the Windows lane; if anything it now produces a real success/failure forwindows-debug/windows-releaseinstead of skipping them outright when lint is red. Worth a quick confirmation against the actual ruleset before merge, since nothing in-repo proves it. - Running two parallel
windows-latestjobs instead of one serial job duplicates per-job fixed overhead (checkout, sccache install, the cache-backend probe, toolchain setup) that was previously paid once — tradeoff. The change's own comments already weigh the clippy/prek-wait removal's cost in wall-clock terms, but not this one in Windows-runner-minutes: total billed Windows compute time may rise somewhat even as end-to-end latency falls. Worth confirming that's an accepted cost, alongside the already-documented cold-cache effect on both jobs' first runs.
Positive signals
ci_windows_aggregator_passes_only_when_every_job_succeededdirectly executes the aggregator's extracted bash script underbash -e -cagainst every result/skip combination (success/failure/cancelled/skipped × expected-skip true/false) rather than only asserting on source text — real behavioral verification of the merge gate, independently re-run and confirmed correct for this review.- Every step from the pre-split monolithic job lands in exactly one of
windows-debug/windows-release, confirmed by diffing the old single job against the new pair step-by-step; nothing was silently dropped or duplicated. - The four supporting files (
docs/testing.md,tests/e2e-cucumber/features/comfyui.feature,xtask/src/env_mutation_contract.rs,.github/workflows/e2e-selfhosted.yml) were each updated to name the correct new job for the behavior they describe — including the non-obvious case that the install-lifecycle/E2E_ALSO_RUN_IDSwork moved towindows-releasewhile the single-processcargo test --workspacemoved towindows-debug, which would have been an easy pair to swap by mistake. ci_windows_jobs_share_one_cache_and_toolchain_setupguards the two duplicated jobs' env and setup steps against silent drift from each other, andci_windows_jobs_keep_every_gating_step_enabledpins every gating step's exact command,if:, and absence ofcontinue-on-error— both checked by direct reading against the literal YAML, including the trickier>-block-scalar steps.- All
uses:lines are pinned to full commit SHAs with trailing version comments, consistent with this repo's own convention; single, clearly-written, DCO-signed commit whose message matches the diff.
Deployment notes
- At head commit
ce854ce9, two self-hosted GPU jobs in the separate "E2E self-hosted" workflow — "E2E tests (Strix Halo, Ubuntu, nightly, current)" and "...release, current)" — show cancelled, and two more ("E2E tests (MI300X, current)", "E2E tests (W7900, current)") remain queued. This PR touches that workflow with only a verified-accurate comment edit, the job logs show an artifact upload immediately followed by orphan-process termination (consistent with a runner-side cancellation rather than a failing assertion), and this repo's ownAGENTS.mddocuments that self-hosted GPU hardware availability can differ from a clean run. All checks this change can actually affect — thewindows-build-and-testaggregator,windows-debug,windows-release,clippy,prek, and the affected-crates test job that compiles/runs the new contract tests — are green. Recommend confirming the self-hosted lanes are unrelated infrastructure flakiness before treating the run as fully clean.
What this covered
Every one of the 6 changed files read completely at the PR's head content (fetched via git show refs/pr810head:<path>, never checked out), plus unchanged repository context (module wiring in xtask/src/main.rs, release.yml/nightly.yml, README.md, other docs) for impact analysis — via two independent blind fan-out workers (one on ci.yml + xtask/src/workflow_contract.rs, one on the four smaller doc/comment files), each scoped to files only, with no access to this PR's description or comments, plus my own consolidation, impact-analysis and design-review pass. Range reviewed: df1548a211ed5a53d37d2e5df8f83a7746a13239…ce854ce901021150f06ba765d8286e0e197a5e9e.
Procedural note: as the orchestrating context I necessarily read this PR's title/body once during the mandatory pre-review duplicate-review check (the API call that returns the head SHA also returns the body); neither fan-out worker saw it, and its content was not used for judgment until after the findings above were settled.
Live CI was read at the exact head commit (gh pr checks): all Windows- and lint-relevant jobs are SUCCESS, as detailed under Deployment notes. "Prior changes" — no earlier review threads exist on this PR to reconcile against (confirmed zero review-comment threads). "History" — compared the pre-split job definition on origin/main against the new pair directly via git show/git diff. Could not execute cargo test -p xtask directly (no Rust toolchain available in this environment); relied on the live CI's own passing "Test (affected crates)" job plus direct re-execution of the aggregator's bash script and manual tracing of the YAML-parsing helpers against the literal file content. Not reconciled against any discussion on this PR.
…vious job The Windows-split rewording left short orphan lines in three prose paragraphs; reflow them with no wording change. ci_jobs() left the comment block above a job on the end of the job before it, so that block carried text that describes its neighbour. Drop the comment and blank lines between jobs, as job_block already does, and pin it with a test that also covers a nested comment on a job's last line. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…step The aggregator tests only looked for `needs.<job>.result` somewhere in the job, so binding the release result to the wrong variable, narrowing the job's `if:`, or adding `if`/`continue-on-error`/`shell` to its step all kept the required check green while letting a failed Windows job through. Assert that each Windows job's result is bound to exactly one variable the script checks, that the job's `if:` is exactly `!cancelled()`, and that it is one plain step on a Linux runner. The script test now takes its variable names from those bindings. The per-job checks cover every job found by runner instead of a fixed pair, with a fixture test for that lookup. Also say in the ci.yml comment that the release side now compiles the e2e harness's dev-profile dependencies itself. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…stale claims The aggregator and gating tests looked for `if` and `continue-on-error` only where a step's keys sit after its first line, so a step written as `- if: false` followed by `name:` passed unseen. Compare the exact key list of the aggregator's job and step instead, which also pins that the job has no `name:` (it would rename the required check), and read gating steps' keys with step_keys. Read the aggregator's env and script through the existing step_env and preflight_body_code instead of new readers. Every step in both Windows jobs must now be in the gating table or on a short list of setup steps, so a step added later cannot go unpinned. Reword four ci.yml comments the split made wrong: the Windows build time in the timeout note, the `windows` output not being a path filter, the heavy-job gating note on the remote-control job, and why the pinned-key check sits in the release job. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…helpers The gating-step test's doc claimed more than it checks: commands are matched by substring and the setup steps are exempt by name, so a flag or a trailing `|| true` passes. Say so, rather than leave a claim the test cannot fail on. Pin the helpers the aggregator tests depend on, so weakening one cannot hide a break in ci.yml: a command that only a comment mentions, a second step in the aggregator, and an env expression bound twice or not at all. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…per tests The step-completeness check accepted any unnamed step whose raw text mentioned one of the two opening actions, comments included, with any keys. Match the `uses:` value, allow only `uses`, `with` and an absent-or-`heavy` `if`, and say in the test's doc what is and is not checked there. The helper tests now also cover a value that only contains the expected expression, and `code_only` keeping one output line per input line. Correct the comment naming which tests use which helper. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…toolchain step by `uses:` The completeness check accepts any number of unnamed checkout or setup-rust-toolchain steps at any position, not only the two that open each job; say so in the doc and inline comments rather than claim more. Find the toolchain step in the shared-setup test by its `uses:` value instead of by substring, so a comment naming the action in an earlier step cannot end the compared region early. Say in code_only's doc that it keeps one output line per input line, which the helper test pins. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
I went through this against the saved diff and found nothing that blocks the merge. I'm requesting changes on the points below, which I think are worth settling first.
Wall-clock gain isn't shown yet. On this PR's own cold-cache run windows-release took 20m31s against 14m46s for windows-debug, so the long pole just moved and it's no better than the old 20+ minute single job for fork PRs. The "about 10 min on main" figure is an estimate for a warm cache, not something measured. Either rebalance the lanes (e.g. move the release-readiness self-test or the harness self-tests across) or re-measure on main before claiming it.
Contract-test cost. About 740 added lines of hand-rolled YAML scraping (indent-based yaml_value_at/ci_jobs, strip_comment) for a two-job split. I understand the required-check risk justifies some of it, but it's brittle against indentation changes and will be a maintenance tax.
Gating test is loose. It matches commands by substring, so a trailing || true on a gating step would still pass. You've disclosed this; I'd tighten it if it's cheap, otherwise leave a note next to the test.
Comment drift. The timeout note still says "roughly 4x the observed p100" while listing Windows at 15-20 min with a 45 min limit (see inline).
Decisions that are yours to make (all disclosed in the PR body, so not asking for changes): dropping the wait on clippy/prek, paying for two Windows runners even on docs-only PRs, and starting both jobs on a cold per-job rust-cache key.
Not checked: I only read the diff, so I didn't grep the rest of the repo (docs, scripts, other workflows) for stale references to the old single Windows job, and I didn't re-run the "broke ci.yml dozens of ways" mutation checks. Worth a grep on your side. CI was still running at this head (the Windows jobs and the aggregator hadn't reported), so this isn't a read on those results.
| @@ -71,17 +71,21 @@ env: | |||
| # with no timeout — holds a runner for six hours and, in the merge queue, stalls | |||
| # every queued PR behind it with no signal that anything is wrong. The values here | |||
| # are deliberately loose (roughly 4x the observed p100, which is under 3 min for | |||
There was a problem hiding this comment.
This note says the limits are "roughly 4x the observed p100", but the Windows entry below it is 15-20 min against a 45 min limit (about 2x). Could you reword so the comment doesn't contradict the value?
There was a problem hiding this comment.
Fair, the wording contradicted the value. Reworded in 0a04bcb: the roughly 4x figure is for most jobs, and the two Windows jobs are called out as the exception at about 2 to 3x (15 to 21 min against 45 min on a cold pull-request cache).
| # of the whole run (it alone decided when CI went green); the debug side | ||
| # (clippy, build, test, smoke) and the release side (release build, install | ||
| # lifecycle, PR bundle) share no outputs, so they run side by side. The one | ||
| # thing the release side gives up is the debug build's warm-up of the e2e |
There was a problem hiding this comment.
This describes the dev-profile warm-up as the only thing the release lane gives up, but on this PR's cold run the release lane took 20m31s versus 14m46s for debug. Consider rebalancing (e.g. move the release-readiness self-test or the harness self-tests) so the two lanes are closer, or note that the gain depends on a warm cache.
There was a problem hiding this comment.
You are right that this overstated it. Reworded in 0a04bcb: the comment now says the release side also gives up the debug build's warm-up, records that on a cold pull-request cache release took about 21 min against about 15 for debug, and says the warm-cache balance has not been measured since the split. I did not rebalance because the movable steps are small (see my reply on the earlier thread); the plan is to compare several push runs on main first. I also dropped a claim in the pinned-key comment that it reuses the release build, since that step took minutes on the cold run and I cannot show from the log that it reuses anything.
The timeout note called every limit roughly 4x the observed p100 while listing a 15-20 min Windows job against a 45 min limit. Say the Windows jobs are the exception. The split comment said the release side gives up only the dev-profile warm-up, but on a cold pull-request cache the release job took about 21 min against about 15 for debug. State that, and that the warm-cache balance is unmeasured. The pinned-key comment claimed it reuses the release build; on a cold cache the step took minutes, so say only what is known. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · e311e0e
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change (base df1548a2…head e311e0e2, 6 files, 977/70 +/-). Purely CI plumbing: splits the Windows GitHub Actions lane into two parallel jobs (windows-debug, windows-release) behind a pinned windows-build-and-test aggregator that keeps the required-check name. No ticket is named in the description, and none appears needed — AGENTS.md §3 exempts "purely internal changes (refactors, CI plumbing, docs)" from a Gherkin scenario provided the PR text says why, which it does.
Blocking
None
Non-blocking
None
Decisions for the author
- The claim that
clippy/prekare already independently-required branch-protection checks is unverifiable from the repo itself — tradeoff.ci.yml's own comment onwindows-debugstates they "stay required checks in their own right, so a lint failure still blocks the merge." I traced GitHub Actions' actualif:semantics (a customif:that doesn't usesuccess()/failure()/cancelled()/always()is still implicitly ANDed withsuccess()of every job inneeds:) and confirmedwindows-debug/windows-release'sif: needs.changes.outputs.windows == 'true'does still requirechangesto succeed, so the mechanism is sound — but whetherclippyandprekare themselves required checks in the branch-protection ruleset (so that dropping them fromneeds:doesn't let a lint failure merge) is a GitHub setting, not something any file here proves. Worth a quick confirmation against the actual ruleset before merge. - Lane rebalancing is an open, self-acknowledged follow-up — tradeoff, not a defect. The PR's own cold-cache run shows
windows-release(20m31s) now the longer pole, versuswindows-debug(14m46s); the "~10 min on main" estimate is explicitly flagged in the PR body as an unmeasured, warm-cache projection. Nothing about the split logic is wrong — gating, aggregation, and step placement all check out (see below) — this is purely a question of whether the wall-clock goal is met yet, which only a warmmainrun (or deliberately moving a self-test across jobs) can answer.
Positive signals
- The
ci_jobs/ci_jobYAML-block-extraction helpers (xtask/src/workflow_contract.rs:4254-4286) correctly drop the comment-and-blank-line region between two job headers rather than attaching it to either neighbor, with a dedicated regression test (ci_jobs_drops_the_comment_between_jobs_instead_of_keeping_it_on_the_job_before) covering a nested end-of-job comment and a trailing-comment job header containing the literal strings!cancelled()/always(). I traced the loop by hand against that fixture and confirmed the boundary logic is sound. ci_windows_aggregator_passes_only_when_every_job_succeededextracts the aggregator's actual bash script from the live YAML and executes it under realbash -e -cacross every (changes, windows-debug, windows-release, expect-skip) combination that matters, rather than only pattern-matching source text — I re-derived the combinatorics by hand and the script's pass/fail verdict is correct in every case I checked, including a failedchangesjob under an expected-skip scenario.- Every GATING-table entry in
ci_windows_jobs_keep_every_gating_step_enabledmatches an actual step in the currentwindows-debug/windows-releaseblocks, and the test's own doc comment accurately discloses its real limitation (substring command matching, name-exempt setup steps) rather than overclaiming — I checked the disclosed gap against the implementation and it is not understated. - The four small supporting files (
docs/testing.md,comfyui.feature,env_mutation_contract.rs,e2e-selfhosted.yml) each attribute the right behavior to the right new job (e.g.E2E_ALSO_RUN_IDStowindows-release, the single-processcargo test --workspacetowindows-debug) and a full-file grep turned up no stale leftover mention of the old single job. - Live CI at this exact head is green on every check the change can affect:
windows-build-and-test,Windows build and test (debug),Windows release build and lifecycle,clippy,prek,Test (affected crates),build-and-test,Commit signatures + sign-off.
Deployment notes
At head e311e0e2, the separate self-hosted "E2E tests" workflow (which this PR touches only via a verified-accurate comment edit) has two GPU lanes still queued/in-progress (MI300X, W7900, and the nightly Strix-Halo-Ubuntu lane); these are optional/non-blocking per that workflow's own code comment ("Non-blocking so it never gates the PR") and unaffected by this change. No action needed before merge on that basis.
What this covered
DEGRADED REVIEW — three parallel fan-out spawns (Agent tool) all failed on a shared, cross-session subagent concurrency limit ("Concurrent subagent limit reached... Do not retry"); ListAgents confirmed 20+ agents running under other sessions on this machine, none mine. Per the review method, this was not retried, and the review proceeded single-context instead, with real correctness tracing (not just reading) to compensate: GitHub Actions' if:/implicit-success() semantics were independently verified against current GitHub docs rather than assumed, the ci_jobs comment-boundary fix was traced by hand against its fixture, and the aggregator's bash script was re-derived combination-by-combination by reasoning (not re-executed, since no local Rust/bash toolchain is available in this environment — the live CI run's own green "Test (affected crates)" job is relied on for actual execution). Recommend re-running with fan-out once subagent capacity is available, for broader coverage than one context can provide.
Read completely: .github/workflows/ci.yml (full file, ~1850 lines, plus its diff), xtask/src/workflow_contract.rs's entire changed region (the Windows-lane tests and the helpers they depend on, old and new) plus the pre-existing helper functions (job_block, strip_comment, indent_of, run_block, step_env, step_keys) those tests reuse, and all four small supporting files in full. Commit-by-commit history of the six follow-up commits since the prior reviewed head was read via their messages (not full git blame). Checked live CI (get_check_runs) and AGENTS.md for this repo's own conventions.
Did not run: "Prior changes" — no applicable prior review thread was reconciled against (per the review method, discussion is read only after findings are settled, and is not referenced here). Could not execute cargo test -p xtask, cargo clippy, or the aggregator's bash script directly (no local toolchain in this sandboxed environment); relied on live CI's passing "Test (affected crates)" job plus manual tracing instead. Not reconciled against any discussion on this PR.
🤖 Generated with Claude Code
|
Thanks for the review. What changed since it, and where I left things: Wall-clock gain. The body now states what has been measured (cold pull-request runs: debug about 14.7 min, release about 20.5 to 21 min, last non-GPU job after about 22 min) and says the gain on main has not been measured since the split. I would rather re-measure over several Comment drift. Both inline comments are fixed in 0a04bcb. I also grepped Gating test is loose. Agreed. It is documented in the test's doc comment (substring match on the command, the setup steps exempt by name), so a trailing Contract-test cost. Fair point. The helpers added since your read reuse the existing Mutation checks. I re-ran them against the current head: each single breakage I tried (a result bound to the wrong job, a narrowed aggregator |
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · 0a04bcb
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change (base df1548a2…head 0a04bcb7, 6 files, +984/-71, 8 commits). Purely CI plumbing: splits the Windows GitHub Actions lane into two parallel jobs (windows-debug, windows-release) behind a pinned windows-build-and-test aggregator that keeps the required-check name. No ticket named; the PR body states this is CI plumbing changing no CLI behaviour, so no Gherkin scenario is needed (AGENTS.md §3's exemption), and the description's stated scope (984/+71 across 6 files, 8 commits) matches the diff exactly.
Blocking
None
Non-blocking
None
Decisions for the author
- The claim that
clippy/prekare already independently-required branch-protection checks is unverifiable from the repo itself — tradeoff.ci.yml's comment onwindows-debug/windows-releasestates they "stay required checks in their own right, so a lint failure still blocks the merge." Theif:mechanism is sound (needs.changes.outputs.windows == 'true'still implicitly requireschangesto succeed), but whetherclippy/prekare themselves required in the branch-protection ruleset is a GitHub setting no file here proves. Worth a quick confirmation against the actual ruleset before merge — this has been raised at each prior head reviewed and still has no in-repo answer, though it is unchanged risk rather than a new one.
Positive signals
- The PR body's "Trade-offs" section discloses exactly what the diff's own
ci.ymlcomments disclose — duplicated per-job fixed overhead (checkout/sccache/toolchain paid twice), the rust-cache key change, the lane-length imbalance (14.7 vs 20.5-21 min cold), and the clippy/prek-wait removal — with no gap between stated and actual tradeoffs. - Every step from the pre-split monolithic Windows job lands in exactly one of
windows-debug/windows-release; confirmed by direct diff of the old single job against the new pair. - The aggregator's bash script (
windows-build-and-test's one step) was traced and executed by hand against the literal YAML across every success/failure/cancelled/skipped × expected-skip combination; the pass/fail verdict is correct in each case, including a failedchangesjob under an expected-skip scenario. - The four small supporting files (
docs/testing.md,comfyui.feature,env_mutation_contract.rs,e2e-selfhosted.yml) each attribute the right post-split behavior to the right job name (e.g.E2E_ALSO_RUN_IDS/install-lifecycle towindows-release,cargo test --workspacetowindows-debug, an easy pair to get backwards), verified against the actual job bodies; a repo-wide grep found no other stale reference to the old single Windows job. - Commit history is 8 small, specifically-worded, DCO-signed commits that progressively tighten the new contract tests (pinning aggregator keys, step keys, gating-step coverage) rather than noise; the final 2 commits touch only comments, confirmed by diffing each commit against
ci.yml.
Deployment notes
Live CI at this exact head is green on all 25 reported checks, including windows-build-and-test, Windows build and test (debug), Windows release build and lifecycle, clippy, prek, Test (affected crates), and Commit signatures + sign-off. No pending or cancelled lanes at this head (unlike earlier heads of this PR, where self-hosted GPU lanes were still queued/cancelled).
What this covered
Every one of the 6 changed files read completely at this head, via two independent blind fan-out workers (one on ci.yml + xtask/src/workflow_contract.rs as a coupled group, one on the four smaller supporting files), each with no access to this PR's description or comments, plus my own direct reading of the Windows job definitions and aggregator logic in ci.yml, a commit-by-commit diff of all 8 commits (confirming the 7 follow-up commits after the initial split touch only xtask tests and comments, never production YAML logic), and a repo-wide grep for stale references to the old single Windows job (none found, aside from an unrelated identically-named windows-release job in the separate release.yml release-publishing workflow). Range reviewed: df1548a211ed5a53d37d2e5df8f83a7746a13239…0a04bcb7e03af879ad4e1602e2f89adc7b49137d.
Live CI was read via get_check_runs at the exact head commit. Could not execute cargo test -p xtask or cargo clippy directly (no Rust toolchain in this sandboxed environment); relied on live CI's own green "Test (affected crates)"/"clippy" jobs plus direct hand-tracing/execution of the aggregator's extracted bash script against the literal YAML instead. "Prior changes" — the same two architectural/tradeoff themes (clippy/prek reliance, lane imbalance) have been raised at each previously-reviewed head of this PR by this same automation and by a human reviewer; both remain author decisions rather than defects, and the lane-imbalance one is now explicitly re-disclosed in this head's own updated comments. Not reconciled against any discussion on this PR.
🤖 Generated with Claude Code
juhovainio
left a comment
There was a problem hiding this comment.
Re-reviewed at 0a04bcb. No blocking findings.
The required windows-build-and-test context is unchanged: the aggregator keeps the job id with no name:, runs under !cancelled(), compares each result by exact equality, and derives its expected skip from the same changes.windows output the work jobs gate on. A failed or skipped changes fails it. Pinned action SHAs are preserved, and all checks are green.
Non-blocking:
- The measured gain is only shown for warm
main; on a cold fork-PR cache the release lane alone (~20-21 min) is about the length of the old single job, so the split mainly saves the clippy/prek wait. - Measured timings hardcoded in
ci.ymlcomments will go stale. - Only
docs/testing.mdis updated; worth a repo-wide grep for other docs that still describewindows-build-and-testas a single job.
For you to decide: dropping the clippy/prek wait costs two Windows runners on lint-red pushes (needs: [changes, prek] is the cheap alternative), and the ~90 lines of duplicated setup across the two jobs is held together by the drift test.
| # lifecycle, PR bundle) share no outputs, so they run side by side. The release | ||
| # side gives up the debug build's warm-up of the e2e harness's dev-profile | ||
| # dependencies, which it now compiles for itself. The two sides are not the same | ||
| # length: on a cold pull-request cache the release job took about 21 min against |
There was a problem hiding this comment.
Non-blocking: these measured durations (about 21 vs 15 min) will go stale in a comment; consider dropping the numbers and pointing at this PR. Also, for cold-cache fork PRs the release lane alone is about the length of the old single job, so the split only helps there via the dropped clippy/prek wait.
There was a problem hiding this comment.
Dropped the minutes from both comments in 95baf4d (this one and the timeout note, which had the same problem); they now say the release job is the longer one on a cold cache and point at the numbers on this pull request. On the cold fork-PR point, the two pre-split cold PR runs I checked had the single Windows job at 1593 s and 1754 s, after waiting about 340 s for the lint jobs. On this pull request's cold runs the release job took 1231 s and 1257 s and started after about 50 s, so the release lane alone is shorter than the old job, and the whole run reached its last non-GPU job about 10 to 13 minutes sooner (about 1320 s against 1950 to 2090 s). It is two data points per side, so I would still confirm on warm main runs after merge.
| # been measured since the split; compare the two jobs' durations over several | ||
| # runs there before rebalancing steps between them. | ||
| # | ||
| # Neither job waits for `clippy` or `prek`. Those stay required checks in their |
There was a problem hiding this comment.
Tradeoff to confirm: neither Windows job waits on clippy/prek any more, so a prek failure (rustfmt, ruff, shellcheck) costs two Windows runners on lint-red pushes. The body offers needs: [changes, prek] (~20 s); worth deciding explicitly.
There was a problem hiding this comment.
Keeping it as it is, since the body discloses the cost and you list it as a decision rather than a defect. For the record, clippy and prek (lint / hygiene) are required checks on this pull request (I read isRequired from GraphQL for the previous head), so a lint failure still blocks the merge. The cost is the two Windows runners on lint-red pushes, and needs: [changes, prek] stays available if you would rather pay about 20 s of wall clock to avoid it.
Minutes quoted in a comment go stale as the runners and caches change. Say which Windows job is the longer one and where to compare the two, without the numbers, in both the timeout note and the split note. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Summary
The Windows build-and-test job is the longest pole of every CI run: 10 to 16 minutes on recent pushes to main, and 20 or more on pull requests from forks, which get no shared compiler cache. It also starts a couple of minutes late because it waits for the Linux
clippyandprekjobs. This splits it into two parallel jobs and removes that wait.Measured so far, on this pull request's cold-cache runs, the two jobs took about 14.7 minutes (
windows-debug) and 20.5 to 21 minutes (windows-release), and the run reached its last non-GPU job after about 22 minutes. The gain on main, where the cache is warm, has not been measured since the split; the estimate from step timings is that Windows drops from about 10 to 16 minutes to roughly the longer of the two lanes, which is not a claim this pull request can verify. It should be checked over severalpushruns on main after merge, comparing the two jobs' durations and the time until the last non-GPU job finishes, before rebalancing steps between the lanes.What changed
windows-debug: Windows clippy, dev build, workspace tests, local smoke, and the harness self-tests.windows-release: release build, install lifecycle e2e, release-readiness self-test, installer syntax check, pinned-key check, and PR bundle packaging. None of these consume the debug side's output. The one thing this side gives up is the debug build's warm-up of the e2e harness's dev-profile dependencies, which it now compiles for itself.windows-build-and-testkeeps its name as the required check and becomes an aggregator that needschanges,windows-debugandwindows-release, and requires each to have finished exactlysuccess. A skip is accepted only for a manual dispatch that did not ask for binaries. The job has noname:of its own, so the check context branch protection matches is unchanged.changes.windowsoutput. Both jobs gate on it and the aggregator derives its expected skip from it, so the two cannot disagree.clippyorprekany more. Both remain required checks of their own, so a lint failure still blocks the merge.sccache stats) runs in both. I checked this by diffing the parsed steps before and after.changesjob's description of its outputs, and the remote-control job's gating note.Why the gate is not weakened
A skipped required check reads as passing, so the aggregator compares each result for exact equality instead of testing for failure:
failure,cancelledand an unexpectedskippedall fail it, and a failedchangesfails it even when the work jobs are meant to be skipped. The contract tests inxtask/src/workflow_contract.rspin this:powershell-lint), bind each job's result to exactly one env variable, and check that variable in its script. The script is executed underbash -eover every combination of job results, using the variable names taken from the workflow's own bindings.runs-on,timeout-minutes,needs,ifandsteps, itsif:is exactly!cancelled(), and it is one step with exactly the keysname,envandrun. Aname:would rename the required check, and anif,continue-on-errororshellwould change when it runs or whether it can fail.needs, runner andif:are pinned exactly with no job-levelcontinue-on-error, and the two jobs' env and setup steps cannot drift apart.if:and set nocontinue-on-error. Every other step in either job must be on a short setup list or be one of the two opening actions, so a step added later cannot go unpinned.What the gating-step test does not check is documented in the test: commands are matched as substrings, so a flag or a trailing
|| trueadded to one would pass, and the setup steps are exempt by name.I broke
ci.ymlin dozens of single ways (dropped steps,continue-on-error,if: false, wrongneeds, a result bound to the wrong job, a narrowed aggregatorif:, aname:on the aggregator, steps keyed on their first line, a step added to either job, a Windows job the aggregator does not await, and so on) and confirmed each one fails a Windows contract test. The helpers the tests rely on have their own unit tests.Trade-offs
clippy/prekwait buys a couple of minutes of wall clock on every run, paid for with Windows runner time on pushes that turn out lint-red. A middle option isneeds: [changes, prek], which costs about 20 seconds; happy to switch if preferred.windows-latestrunners start instead of one, including on a docs-only pull request, where the build, test and check steps are skipped because they are gated onheavyand the jobs do little beyond setup. Fixed per-job overhead (checkout, sccache install, the cache probe, toolchain setup) is paid twice, so billed Windows time can rise while wall clock falls.windows-build-and-testcache entry is orphaned until it ages out, and the Actions cache now holds two Windows lanes where it held one. The sccache prefix is unchanged, so the compiler cache stays warm.cargo xtask e2ebuilds the e2e harness in the dev profile, which now happens inwindows-releaseinstead of being warmed by the debug build. sccache should absorb this after the first run.Verification
cargo test -p xtask(353 tests),cargo fmt --checkandcargo clippy -p xtask --all-targets -- -D warningspass locally, and the leak scan and conflict-marker grep are clean.windows-debug,windows-releaseandwindows-build-and-test. The commits added since then change onlyxtasktest code and comments inci.ymland the docs, and their CI run is the first check of those.actionlintwas not available locally.This is CI plumbing and changes no CLI behaviour, so there is no Gherkin scenario.