Skip to content

ci(nightly): consolidate the nightly E2E lanes into one report - #221

Merged
volen-silo merged 2 commits into
mainfrom
feat/nightly-consolidated-report
Aug 12, 2026
Merged

volen-silo merged 2 commits into
mainfrom
feat/nightly-consolidated-report

Conversation

@volen-silo

@volen-silo volen-silo commented Aug 12, 2026 •

Copy link
Copy Markdown
Collaborator
  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. Not applicable — CI-only change.

Summary

The three nightly E2E lanes each uploaded a report nobody joined up, so reading a nightly meant downloading three artifacts and comparing them by hand. That is the run most worth reading: only nightly sets E2E_INCLUDE_NIGHTLY, so it covers scenarios no per-PR run does.

It now produces the same consolidated grid the per-PR self-hosted workflow already has — the one at the bottom of an E2E self-hosted run summary.

The job runs unconditionally rather than behind a paths filter: there is none to consult on a schedule, and the lanes are continue-on-error, so a failed lane must still be reported. A nightly that broke is exactly when the grid is worth having.

The rename is the load-bearing part

The three artifacts are renamed to the canonical e2e-gpu-report / e2e-gpu-strix-ubuntu-report / e2e-gpu-strix-windows-report.

This is not tidying. parse_descriptor in the e2e-report crate maps the artifact name to Platform/OS, and an unrecognised name falls through to fallback_descriptor, which titlecases the name and hardcodes the OS to Linux. So e2e-gpu-nightly-strix-windows-report would have produced a row reading "Gpu Nightly Strix Windows / Linux" — the Windows lane reported as Linux. lib.rs:418 already warns about exactly this hazard for a different case.

Nothing outside nightly.yml referenced the old names (checked), and artifacts are scoped to their run, so sharing names with the per-PR workflow collides with nothing.

Test plan

Two tests, because the failure mode is a silently wrong label rather than a broken build:

  • every e2e artifact any workflow uploads must carry a name the report can map — scans ci.yml, e2e-selfhosted.yml and nightly.yml;
  • the nightly lanes must publish the same platforms as the per-PR lanes — if those diverge, the two grids compare different hardware without saying so.

Both were verified to fail against the old name, not merely to pass now:

nightly.yml uploads `e2e-gpu-nightly-strix-windows-report`, which the report cannot
map to a platform; it would render as a guessed name on Linux. Use one of
["e2e-report", "e2e-gpu-report", "e2e-gpu-strix-ubuntu-report",
 "e2e-gpu-strix-windows-report"] or teach parse_descriptor.

cargo test -p xtask: 74 pass. cargo fmt --all --check, clippy -p xtask --all-targets -D warnings, and the prek hooks including check-yaml are clean.

Verification gap: the job itself cannot run until this is on main and the schedule fires (or someone dispatches the workflow). What is verified here is the naming contract and the YAML; that the consolidated step renders is inherited from the identical per-PR job, which runs on every heavy PR. Worth a workflow_dispatch after merge to confirm end to end.


Review follow-up (44076e2)

  • Doc comment on CANONICAL_REPORT_ARTIFACTS corrected. It claimed e2e-unknown-report was unrecognised and would take the Linux default; it is recognised, and maps to Unknown/Unknown precisely so it escapes that default. Its omission from the const is still right — it is a runtime-only sentinel no workflow can name — and the comment now says that. Also stopped describing label_for_root_report as recognising artifact names; it emits one from a platform slug.
  • Both stale docs updated — docs/ci-hardware-testing.md and tests/e2e-cucumber/README.md each enumerated two report jobs and asserted both were named e2e-report. This PR made both statements false, so it should fix them.
  • Guard test widened on both axes. It now globs .github/workflows rather than listing three files, and collects every e2e--prefixed artifact rather than only -report ones — the latter was the same shape it then asserted, so the population was defined by the assertion. Verified by breaking it two ways: a -nightly- name and an e2e-gpu-results name both now fail, and the second would have passed before. This incidentally makes the consolidated-output filters live; they could not match anything previously.
  • permissions: contents: read added to the job, which was inheriting contents: write from the workflow block that exists for the release jobs.
  • AI-generation footer removed, as requested.

On the duplication — deferring, deliberately

Declining for this PR, with reasons rather than by omission. Extracting a composite action would mean rewriting the two working report jobs as well as adding the third, so a subtle mistake takes out the per-PR report that runs on every heavy PR — a strictly larger blast radius than the change being asked for. It also introduces the repo's first composite action as a side effect of a nightly-report PR, which is the wrong place to decide that pattern.

The cost you name is real and I am not disputing it: four SHA pins now maintained in twelve places, and a flattening explainer written five ways. Three copies is the usual threshold and this is the third. I would rather see it done as its own change, where the diff is legible as a refactor and a reviewer can judge the composite action on its merits.

On #141

Confirmed independently. Once both land, e2e-gpu-strix-wsl-report fails the naming guard (gpu-strix-wsl is not a parse_descriptor arm) and the set-equality test, since nightly has no matching lane. I read that as the invariants working rather than being too strict — #141 needs a parse_descriptor arm regardless, or that lane renders as "Gpu Strix Wsl / Linux" in its own report today. Whoever lands second will need to sequence it.

The three nightly lanes each uploaded a report nobody joined up, so reading a
nightly meant downloading three artifacts and comparing them by hand. That is
the run worth reading most: only nightly sets E2E_INCLUDE_NIGHTLY, so it covers
scenarios no per-PR run does. It now gets the same consolidated grid the per-PR
self-hosted workflow has had.

The job runs unconditionally rather than behind a paths filter: there is none to
consult on a schedule, and the lanes are continue-on-error, so a failed lane
must still be reported. A nightly that broke is exactly when the grid is worth
having.

The artifacts are renamed to the canonical `e2e-gpu[-strix-…]-report` names.
This is not cosmetic. `parse_descriptor` maps artifact names to Platform/OS, and
an unrecognised name falls through to a titlecased platform with the OS
hardcoded to Linux -- so `e2e-gpu-nightly-strix-windows-report` would have
rendered the Windows lane as Linux. Names are scoped to their run, so sharing
them with the per-PR workflow collides with nothing.

Two tests guard it, because the failure mode is a silently wrong label rather
than a broken build: every e2e artifact any workflow uploads must carry a name
the report can map, and the nightly lanes must publish the same platforms as the
per-PR lanes -- if those diverge, the two grids compare different hardware
without saying so. Both were checked against the old name and do fail on it.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
@volen-silo
volen-silo requested a review from a team as a code owner August 12, 2026 11:59
@fredespi

Copy link
Copy Markdown
Collaborator

Reviewed at bd6d160 (diff against 63639db, 2 files, +125/-3). Nothing blocking — recommendation at the end, plus two notes worth having on the record.

The rename really is load-bearing, and I checked the mechanism rather than taking the description's word for it. parse_descriptor matches on the core between e2e- and -report, so the old e2e-gpu-nightly-strix-windows-report reduced to gpu-nightly-strix-windows, missed every arm, and fell through to fallback_descriptor — which titlecases the core and hardcodes Linux. The Windows lane would have rendered as "Gpu Nightly Strix Windows / Linux" in the new grid. Renaming to the canonical names is what makes the consolidated report correct rather than merely present, exactly as you describe.

The rename is complete and safe. No reference to any of the three old names survives anywhere in the tree, and nothing downloads artifacts across workflow runs (no run-id/github-token usage), so nothing outside nightly.yml could have been depending on them. Sharing names with e2e-selfhosted.yml is fine since artifacts are run-scoped.

The *-report download pattern is safe here. nightly.yml has exactly four upload-artifact steps, and the only one the glob could wrongly pull in — the consolidated output — is named e2e-consolidated-report-nightly, which doesn't end in -report. No stray artifact in that workflow can contaminate the grid.

The new job is a faithful copy of the per-PR one, with the changes/serve gate correctly replaced by a plain always(). Your justification holds: there's no paths filter on a schedule, and with continue-on-error lanes a failed nightly is precisely when you want the grid.

Two things I'd raise, neither blocking.

The guard test scans a hardcoded ["ci.yml", "e2e-selfhosted.yml", "nightly.yml"], but its name and comment say "every e2e artifact any workflow uploads". Those are currently the same set — I checked every workflow in the tree and only those three carry e2e-*-report uploads — so the test is complete today. It just won't stay complete on its own: a future e2e-<something>.yml would upload under a new name and slip past the very net this PR adds, reintroducing the silent-Linux mislabel. Globbing .github/workflows/*.yml instead of listing three files would close that without changing anything else.

The second is a heads-up rather than a criticism. the_nightly_lanes_publish_the_same_platforms_as_the_per_pr_lanes asserts strict set equality, and #141 adds e2e-gpu-strix-wsl-report to e2e-selfhosted.yml only. Once both land, that test fails until the nightly workflow grows a matching WSL lane — and the naming guard will fail on that artifact too, since gpu-strix-wsl isn't a parse_descriptor arm. I think that is the invariant doing its job rather than being too strict, but whoever lands the two will need to sequence it deliberately, and it's cheaper to know now than to hit it as a red build.

Recommendation: ready to approve. The naming contract and YAML are verified, and the consolidation step itself is inherited from the per-PR job that runs on every heavy PR. Worth being explicit that this is a diff review only — as you note, the job can't actually execute until it's on main and the schedule fires or someone dispatches it, so the workflow_dispatch you suggest afterwards is the real end-to-end confirmation. Also worth knowing this is currently the only review on the PR.

@juhovainio

Copy link
Copy Markdown
Collaborator

Reviewed against the live branch (head bd6d160). Small, well-scoped CI change and I didn't find anything to flag.

Checked the load-bearing claim directly against crates/e2e-report/src/lib.rs's parse_descriptor: the four canonical names (e2e-report, e2e-gpu-report, e2e-gpu-strix-ubuntu-report, e2e-gpu-strix-windows-report) map to exactly the Platform/OS pairs the PR describes, and that function's own test table confirms it. So the rename genuinely fixes a real mislabeling risk (Windows lane silently rendering as Linux via fallback_descriptor), not just cosmetics.

Also verified:

  • The new e2e-report-nightly job's needs: list (e2e-gpu-nightly, e2e-gpu-nightly-strix, e2e-gpu-nightly-strix-windows) matches the actual job IDs in the file.
  • No other artifact in nightly.yml matches the *-report download glob besides the three renamed lane reports — the new job's own e2e-consolidated-report-nightly output is uploaded after the download step runs, so there's no self-collision.
  • Grepped all three workflow files (ci.yml, e2e-selfhosted.yml, nightly.yml) for every name: e2e... artifact upload; the new every_uploaded_e2e_artifact_has_a_name_the_report_can_label test's line-scanning approach doesn't produce false positives from job/step display names (those are capitalized E2E ..., so the lowercase e2e- prefix check correctly excludes them) and doesn't miss any real upload.
  • Action pins (checkout, setup-rust-toolchain, download-artifact, upload-artifact) in the new job are byte-for-byte the same SHAs as the identical per-PR job in e2e-selfhosted.yml.
  • Confirmed nothing else in the tree referenced the old -nightly- artifact names before this PR removed them.

All CI is green, and per the PR's own honestly-stated verification gap, the new job itself can't be exercised until the schedule fires (or someone runs workflow_dispatch) — that's a real gap a diff review can't close, not something this review can wave away.

Looks safe to approve.

rominf
rominf previously requested changes Aug 12, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed with fredespi that the rename is load-bearing rather than tidying, and that it's complete. Three things not yet raised: two doc-accuracy problems this PR introduces, and the duplication question.

The CANONICAL_REPORT_ARTIFACTS doc comment is wrong (xtask/src/e2e_report.rs:146-148)

Artifact names label_for_root_report and the e2e-report crate's parse_descriptor both recognise. Anything else falls back to a titlecased platform with the OS hardcoded to Linux.

Both halves are false for e2e-unknown-report. It is recognised by both (crates/e2e-report/src/lib.rs:422, xtask/src/e2e_report.rs:138) yet is absent from the const; and lib.rs:417-421 deliberately carves it out to Unknown/Unknown precisely so it does not take the Linux default — the exact hazard this PR exists to fix.

The omission itself is correct, since that name is emitted only at runtime and never appears in .github/. It's the comment that needs to say so. Stating that the const is the set of names a workflow may statically upload, with e2e-unknown-report excluded as a runtime-only sentinel, is about a three-line fix.

(Third nit in the same comment: "label_for_root_report … recognise[s]" conflates producer and consumer — that function takes no artifact name, it reads platform_slug and emits one.)

Two docs are made stale and neither is updated

  • docs/ci-hardware-testing.md:84-86 — "Each workflow has its own consolidated report job (both named e2e-report internally)" followed by an enumeration of two. There are now three, and the third is e2e-report-nightly, so both the count and the "both named e2e-report" claim are wrong.
  • tests/e2e-cucumber/README.md:144-146 — the same two-item enumeration, and the paragraph immediately below describes nightly's three lanes without mentioning that nightly now consolidates them too.

Duplication — non-blocking, but worth a deliberate decision now

Stripping comments, the new job's executable YAML is byte-identical to both existing copies apart from job id, display name, needs, if, and the output artifact name:

  • .github/workflows/ci.yml:724
  • .github/workflows/e2e-selfhosted.yml:646
  • .github/workflows/nightly.yml:634

Same four steps, same four action SHA pins — now maintained in twelve places. AGENTS.md:143 requires the SHA and its version comment to move together, so Dependabot must land three identical edits per action bump. The download-artifact@v8 flattening explainer is now written five times in three different wordings (ci.yml:743, e2e-selfhosted.yml:667, nightly.yml:652, docs/ci-hardware-testing.md:131, xtask/src/e2e_report.rs:64); a comment that has to be kept in sync five ways is its own smell.

A composite action at .github/actions/e2e-consolidated-report/action.yml taking one artifact-name input would leave each job with only its genuinely per-workflow parts (runs-on, needs, if) plus a single uses:. In fairness the repo has no composite actions or reusable workflows today, so this PR would be creating that infrastructure rather than declining to use existing infrastructure — a legitimate reason to defer. But three copies is the usual threshold, and the pin-maintenance cost is concrete.

Smaller

  • The PR body carries an AI-generation footer — worth removing on a public repo.
  • uploaded_report_artifacts (xtask/src/e2e_report.rs:157) collects only names matching e2e-*-report and then asserts they're canonical, so the population is defined by the same shape being asserted. An artifact named e2e-gpu-nightly-results is invisible to it, despite the test name promising "every uploaded e2e artifact". Widening the filter to starts_with("e2e-") closes that, and incidentally makes the currently-inert retain-filters at :204-205 live — neither e2e-consolidated-report-nightly nor -selfhosted ends in -report, so today they can never match anything.
  • nightly.yml:634 inherits contents: write from the workflow-level block set for the release-publishing jobs. Both siblings run under contents: read, and this job only checks out, downloads artifacts, runs an xtask and uploads.

On the #141 heads-up — confirmed. #141 adds e2e-gpu-strix-wsl-report to e2e-selfhosted.yml only, and gpu-strix-wsl is not a parse_descriptor arm, so once both land the naming guard fails on that artifact and the set-equality test fails because nightly has no matching lane. Worth sequencing deliberately. Independently of this PR, #141 probably wants a parse_descriptor arm anyway: without one that artifact currently renders as "Gpu Strix Wsl / Linux".

The const's doc comment claimed `e2e-unknown-report` was unrecognised and
would take the Linux default. Both halves were wrong: it is recognised, and
`parse_descriptor` maps it to Unknown/Unknown precisely so it escapes that
default -- the hazard this change exists to prevent. Its absence from the const
is still right, because it is a runtime-only sentinel no workflow can name; the
comment now says that instead of the opposite. Also stopped describing
`label_for_root_report` as recognising artifact names, since it emits one from a
platform slug rather than parsing one.

Two docs enumerated the two consolidated report jobs and asserted both were
named `e2e-report` internally. There are now three, and the third is not, so
both statements were made false here rather than merely left behind.

The guard test scanned three hardcoded workflows and collected only names ending
in `-report` -- the same shape it then asserted, so the population was defined by
the assertion and an `e2e-gpu-results` was invisible to a test claiming to cover
every e2e artifact. It now globs `.github/workflows` and collects every `e2e-`
prefixed artifact, which also makes the consolidated-output filters live rather
than inert. Both new failure modes were verified to fail, not merely to pass.

The job also inherited `contents: write` from the workflow block that exists for
the release jobs. It checks out, downloads, runs an xtask and uploads.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Addressed in 44076e2. Point by point, including the one I did not take.

The CANONICAL_REPORT_ARTIFACTS doc comment — you were right, and it was worse than imprecise. It asserted the opposite of the truth: e2e-unknown-report is recognised by both, and parse_descriptor maps it to Unknown/Unknown specifically so it escapes the Linux default — the very hazard this PR exists to prevent. The comment now states that the const is the set of names a workflow may statically upload, with that name excluded as a runtime-only sentinel no workflow can name. Also dropped the producer/consumer conflation: label_for_root_report emits a name from a platform slug, it does not recognise one.

Both docs updated. docs/ci-hardware-testing.md and tests/e2e-cucumber/README.md each enumerated two report jobs and asserted both were named e2e-report internally. This PR made those statements false, so fixing them belongs here. The former also now records why the artifact names are canonical, so the next person doesn't have to rediscover the mislabelling mechanism from the test failure.

The guard test was weaker than its own name, on both axes. It scanned three hardcoded workflows (your and @fredespi's point) and collected only -report names before asserting they were canonical — so the population was defined by the same shape being asserted, and e2e-gpu-results was invisible. It now globs .github/workflows and collects every e2e--prefixed name.

Verified by breaking it two ways rather than by re-reading it:

nightly.yml uploads `e2e-gpu-nightly-strix-windows-report`, which the report cannot map to a platform…
nightly.yml uploads `e2e-gpu-results`, which the report cannot map to a platform…

The second would have passed before. As you noted, this also makes the e2e-consolidated-report* filters live — they could not previously match anything.

permissions: contents: read added. Correct catch: it was inheriting contents: write from the workflow-level block that exists for the release-publishing jobs, and it only checks out, downloads, runs an xtask and uploads.


The duplication — declined for this PR, and I want that visible rather than quietly skipped.

Your reasoning holds: three copies is the threshold, twelve pin sites is a real maintenance cost, and the flattening explainer now exists in five wordings. I'm not disputing any of it.

I'm declining because extracting a composite action means rewriting the two working report jobs as well as adding the third. A subtle error there takes out the per-PR report that runs on every heavy PR — a strictly larger blast radius than the change being asked for. It would also make a nightly-report PR the place where this repo's first composite action gets decided, which is the wrong venue for that call.

It's worth doing as its own change, where the diff reads as a refactor and the composite action can be judged on its merits.

On #141 — confirmed independently. Once both land, e2e-gpu-strix-wsl-report fails the naming guard (gpu-strix-wsl is not a parse_descriptor arm) and the set-equality test, since nightly has no matching lane. That reads to me as the invariants working. Worth noting #141 wants a parse_descriptor arm regardless of this PR — without one, that lane renders as "Gpu Strix Wsl / Linux" in its own report today.

The AI-generation footer is gone from the description.

@volen-silo
volen-silo dismissed rominf’s stale review August 12, 2026 15:16

All blocking points addressed in 44076e2: the CANONICAL_REPORT_ARTIFACTS doc comment (which asserted the opposite of the truth about e2e-unknown-report), both stale docs, the guard test widened on both axes and verified to fail two ways, and permissions narrowed to contents: read.

The duplication point is declined rather than addressed, deliberately and visibly — extracting a composite action would mean rewriting the two working report jobs, a larger blast radius than this change, and would decide the repo's first composite action inside a nightly-report PR. Reasoning in the PR comment; worth doing as its own change.

Dismissing to unblock, not to close the discussion. Happy to reopen if the duplication call is wrong.

@volen-silo
volen-silo requested a review from rominf August 12, 2026 15:19
@volen-silo
volen-silo added this pull request to the merge queue Aug 12, 2026
Merged via the queue into main with commit db2c6ab Aug 12, 2026
21 of 22 checks passed
@volen-silo
volen-silo deleted the feat/nightly-consolidated-report branch August 12, 2026 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants