Repository navigation
ci(nightly): consolidate the nightly E2E lanes into one report - #221
Conversation
The three nightly lanes each uploaded a report nobody joined up, so reading a nightly meant downloading three artifacts and comparing them by hand. That is the run worth reading most: only nightly sets E2E_INCLUDE_NIGHTLY, so it covers scenarios no per-PR run does. It now gets the same consolidated grid the per-PR self-hosted workflow has had. The job runs unconditionally rather than behind a paths filter: there is none to consult on a schedule, and the lanes are continue-on-error, so a failed lane must still be reported. A nightly that broke is exactly when the grid is worth having. The artifacts are renamed to the canonical `e2e-gpu[-strix-…]-report` names. This is not cosmetic. `parse_descriptor` maps artifact names to Platform/OS, and an unrecognised name falls through to a titlecased platform with the OS hardcoded to Linux -- so `e2e-gpu-nightly-strix-windows-report` would have rendered the Windows lane as Linux. Names are scoped to their run, so sharing them with the per-PR workflow collides with nothing. Two tests guard it, because the failure mode is a silently wrong label rather than a broken build: every e2e artifact any workflow uploads must carry a name the report can map, and the nightly lanes must publish the same platforms as the per-PR lanes -- if those diverge, the two grids compare different hardware without saying so. Both were checked against the old name and do fail on it. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
|
Reviewed at The rename really is load-bearing, and I checked the mechanism rather than taking the description's word for it. The rename is complete and safe. No reference to any of the three old names survives anywhere in the tree, and nothing downloads artifacts across workflow runs (no The The new job is a faithful copy of the per-PR one, with the Two things I'd raise, neither blocking. The guard test scans a hardcoded The second is a heads-up rather than a criticism. Recommendation: ready to approve. The naming contract and YAML are verified, and the consolidation step itself is inherited from the per-PR job that runs on every heavy PR. Worth being explicit that this is a diff review only — as you note, the job can't actually execute until it's on |
|
Reviewed against the live branch (head Checked the load-bearing claim directly against Also verified:
All CI is green, and per the PR's own honestly-stated verification gap, the new job itself can't be exercised until the schedule fires (or someone runs Looks safe to approve. |
rominf
left a comment
There was a problem hiding this comment.
Agreed with fredespi that the rename is load-bearing rather than tidying, and that it's complete. Three things not yet raised: two doc-accuracy problems this PR introduces, and the duplication question.
The CANONICAL_REPORT_ARTIFACTS doc comment is wrong (xtask/src/e2e_report.rs:146-148)
Artifact names
label_for_root_reportand the e2e-report crate'sparse_descriptorboth recognise. Anything else falls back to a titlecased platform with the OS hardcoded to Linux.
Both halves are false for e2e-unknown-report. It is recognised by both (crates/e2e-report/src/lib.rs:422, xtask/src/e2e_report.rs:138) yet is absent from the const; and lib.rs:417-421 deliberately carves it out to Unknown/Unknown precisely so it does not take the Linux default — the exact hazard this PR exists to fix.
The omission itself is correct, since that name is emitted only at runtime and never appears in .github/. It's the comment that needs to say so. Stating that the const is the set of names a workflow may statically upload, with e2e-unknown-report excluded as a runtime-only sentinel, is about a three-line fix.
(Third nit in the same comment: "label_for_root_report … recognise[s]" conflates producer and consumer — that function takes no artifact name, it reads platform_slug and emits one.)
Two docs are made stale and neither is updated
docs/ci-hardware-testing.md:84-86— "Each workflow has its own consolidated report job (both namede2e-reportinternally)" followed by an enumeration of two. There are now three, and the third ise2e-report-nightly, so both the count and the "both namede2e-report" claim are wrong.tests/e2e-cucumber/README.md:144-146— the same two-item enumeration, and the paragraph immediately below describes nightly's three lanes without mentioning that nightly now consolidates them too.
Duplication — non-blocking, but worth a deliberate decision now
Stripping comments, the new job's executable YAML is byte-identical to both existing copies apart from job id, display name, needs, if, and the output artifact name:
.github/workflows/ci.yml:724.github/workflows/e2e-selfhosted.yml:646.github/workflows/nightly.yml:634
Same four steps, same four action SHA pins — now maintained in twelve places. AGENTS.md:143 requires the SHA and its version comment to move together, so Dependabot must land three identical edits per action bump. The download-artifact@v8 flattening explainer is now written five times in three different wordings (ci.yml:743, e2e-selfhosted.yml:667, nightly.yml:652, docs/ci-hardware-testing.md:131, xtask/src/e2e_report.rs:64); a comment that has to be kept in sync five ways is its own smell.
A composite action at .github/actions/e2e-consolidated-report/action.yml taking one artifact-name input would leave each job with only its genuinely per-workflow parts (runs-on, needs, if) plus a single uses:. In fairness the repo has no composite actions or reusable workflows today, so this PR would be creating that infrastructure rather than declining to use existing infrastructure — a legitimate reason to defer. But three copies is the usual threshold, and the pin-maintenance cost is concrete.
Smaller
- The PR body carries an AI-generation footer — worth removing on a public repo.
uploaded_report_artifacts(xtask/src/e2e_report.rs:157) collects only names matchinge2e-*-reportand then asserts they're canonical, so the population is defined by the same shape being asserted. An artifact namede2e-gpu-nightly-resultsis invisible to it, despite the test name promising "every uploaded e2e artifact". Widening the filter tostarts_with("e2e-")closes that, and incidentally makes the currently-inert retain-filters at:204-205live — neithere2e-consolidated-report-nightlynor-selfhostedends in-report, so today they can never match anything.nightly.yml:634inheritscontents: writefrom the workflow-level block set for the release-publishing jobs. Both siblings run undercontents: read, and this job only checks out, downloads artifacts, runs an xtask and uploads.
On the #141 heads-up — confirmed. #141 adds e2e-gpu-strix-wsl-report to e2e-selfhosted.yml only, and gpu-strix-wsl is not a parse_descriptor arm, so once both land the naming guard fails on that artifact and the set-equality test fails because nightly has no matching lane. Worth sequencing deliberately. Independently of this PR, #141 probably wants a parse_descriptor arm anyway: without one that artifact currently renders as "Gpu Strix Wsl / Linux".
The const's doc comment claimed `e2e-unknown-report` was unrecognised and would take the Linux default. Both halves were wrong: it is recognised, and `parse_descriptor` maps it to Unknown/Unknown precisely so it escapes that default -- the hazard this change exists to prevent. Its absence from the const is still right, because it is a runtime-only sentinel no workflow can name; the comment now says that instead of the opposite. Also stopped describing `label_for_root_report` as recognising artifact names, since it emits one from a platform slug rather than parsing one. Two docs enumerated the two consolidated report jobs and asserted both were named `e2e-report` internally. There are now three, and the third is not, so both statements were made false here rather than merely left behind. The guard test scanned three hardcoded workflows and collected only names ending in `-report` -- the same shape it then asserted, so the population was defined by the assertion and an `e2e-gpu-results` was invisible to a test claiming to cover every e2e artifact. It now globs `.github/workflows` and collects every `e2e-` prefixed artifact, which also makes the consolidated-output filters live rather than inert. Both new failure modes were verified to fail, not merely to pass. The job also inherited `contents: write` from the workflow block that exists for the release jobs. It checks out, downloads, runs an xtask and uploads. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
|
Addressed in The Both docs updated. The guard test was weaker than its own name, on both axes. It scanned three hardcoded workflows (your and @fredespi's point) and collected only Verified by breaking it two ways rather than by re-reading it: The second would have passed before. As you noted, this also makes the
The duplication — declined for this PR, and I want that visible rather than quietly skipped. Your reasoning holds: three copies is the threshold, twelve pin sites is a real maintenance cost, and the flattening explainer now exists in five wordings. I'm not disputing any of it. I'm declining because extracting a composite action means rewriting the two working report jobs as well as adding the third. A subtle error there takes out the per-PR report that runs on every heavy PR — a strictly larger blast radius than the change being asked for. It would also make a nightly-report PR the place where this repo's first composite action gets decided, which is the wrong venue for that call. It's worth doing as its own change, where the diff reads as a refactor and the composite action can be judged on its merits. On #141 — confirmed independently. Once both land, The AI-generation footer is gone from the description. |
All blocking points addressed in 44076e2: the CANONICAL_REPORT_ARTIFACTS doc comment (which asserted the opposite of the truth about e2e-unknown-report), both stale docs, the guard test widened on both axes and verified to fail two ways, and permissions narrowed to contents: read.
The duplication point is declined rather than addressed, deliberately and visibly — extracting a composite action would mean rewriting the two working report jobs, a larger blast radius than this change, and would decide the repo's first composite action inside a nightly-report PR. Reasoning in the PR comment; worth doing as its own change.
Dismissing to unblock, not to close the discussion. Happy to reopen if the duplication call is wrong.
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. Not applicable — CI-only change.Summary
The three nightly E2E lanes each uploaded a report nobody joined up, so reading a nightly meant downloading three artifacts and comparing them by hand. That is the run most worth reading: only nightly sets
E2E_INCLUDE_NIGHTLY, so it covers scenarios no per-PR run does.It now produces the same consolidated grid the per-PR self-hosted workflow already has — the one at the bottom of an
E2E self-hostedrun summary.The job runs unconditionally rather than behind a paths filter: there is none to consult on a schedule, and the lanes are
continue-on-error, so a failed lane must still be reported. A nightly that broke is exactly when the grid is worth having.The rename is the load-bearing part
The three artifacts are renamed to the canonical
e2e-gpu-report/e2e-gpu-strix-ubuntu-report/e2e-gpu-strix-windows-report.This is not tidying.
parse_descriptorin the e2e-report crate maps the artifact name to Platform/OS, and an unrecognised name falls through tofallback_descriptor, which titlecases the name and hardcodes the OS to Linux. Soe2e-gpu-nightly-strix-windows-reportwould have produced a row reading "Gpu Nightly Strix Windows / Linux" — the Windows lane reported as Linux.lib.rs:418already warns about exactly this hazard for a different case.Nothing outside
nightly.ymlreferenced the old names (checked), and artifacts are scoped to their run, so sharing names with the per-PR workflow collides with nothing.Test plan
Two tests, because the failure mode is a silently wrong label rather than a broken build:
ci.yml,e2e-selfhosted.ymlandnightly.yml;Both were verified to fail against the old name, not merely to pass now:
cargo test -p xtask: 74 pass.cargo fmt --all --check,clippy -p xtask --all-targets -D warnings, and the prek hooks includingcheck-yamlare clean.Verification gap: the job itself cannot run until this is on
mainand the schedule fires (or someone dispatches the workflow). What is verified here is the naming contract and the YAML; that the consolidated step renders is inherited from the identical per-PR job, which runs on every heavy PR. Worth aworkflow_dispatchafter merge to confirm end to end.Review follow-up (44076e2)
CANONICAL_REPORT_ARTIFACTScorrected. It claimede2e-unknown-reportwas unrecognised and would take the Linux default; it is recognised, and maps to Unknown/Unknown precisely so it escapes that default. Its omission from the const is still right — it is a runtime-only sentinel no workflow can name — and the comment now says that. Also stopped describinglabel_for_root_reportas recognising artifact names; it emits one from a platform slug.docs/ci-hardware-testing.mdandtests/e2e-cucumber/README.mdeach enumerated two report jobs and asserted both were namede2e-report. This PR made both statements false, so it should fix them..github/workflowsrather than listing three files, and collects everye2e--prefixed artifact rather than only-reportones — the latter was the same shape it then asserted, so the population was defined by the assertion. Verified by breaking it two ways: a-nightly-name and ane2e-gpu-resultsname both now fail, and the second would have passed before. This incidentally makes the consolidated-output filters live; they could not match anything previously.permissions: contents: readadded to the job, which was inheritingcontents: writefrom the workflow block that exists for the release jobs.On the duplication — deferring, deliberately
Declining for this PR, with reasons rather than by omission. Extracting a composite action would mean rewriting the two working report jobs as well as adding the third, so a subtle mistake takes out the per-PR report that runs on every heavy PR — a strictly larger blast radius than the change being asked for. It also introduces the repo's first composite action as a side effect of a nightly-report PR, which is the wrong place to decide that pattern.
The cost you name is real and I am not disputing it: four SHA pins now maintained in twelve places, and a flattening explainer written five ways. Three copies is the usual threshold and this is the third. I would rather see it done as its own change, where the diff is legible as a refactor and a reviewer can judge the composite action on its merits.
On #141
Confirmed independently. Once both land,
e2e-gpu-strix-wsl-reportfails the naming guard (gpu-strix-wslis not aparse_descriptorarm) and the set-equality test, since nightly has no matching lane. I read that as the invariants working rather than being too strict — #141 needs aparse_descriptorarm regardless, or that lane renders as "Gpu Strix Wsl / Linux" in its own report today. Whoever lands second will need to sequence it.