Repository navigation
Conversation
juhovainio
left a comment
There was a problem hiding this comment.
Thanks for tracking this down. I went through the CI logs on this head, and two things need fixing before merge.
1. The WSL2 E2E job regresses because of this change. Scenario runtime-09 (@id:runtime-resolve-canonical-release) fails: it expects the preview source https://repo.amd.com/rocm/whl-multi-arch but gets the runner's mirror URL, because ROCM_CLI_THEROCK_RELEASE_PIP_BASE is now forwarded into the guest. WSL2 passed on the recent main runs and fails only on this PR (run 41531206186, 1 unexpected failure; the other WSL2 failures are expected xfails). The Windows lane fails with the identical assertion, but it is already red on main and this PR doesn't touch it. Either the scenario/harness (isolate_env()) should clear or override the TheRock base variables when asserting canonical provenance, or only the HF_* names should be forwarded.
2. The commit has no Signed-off-by trailer and isn't GitHub-Verified, so the "Commit signatures + sign-off" check fails. Please amend with git commit -s (signing enabled) and re-push.
Smaller points:
- The name list is now duplicated between
e2e-selfhosted.ymlandnightly.yml(nightly also still lacksE2E_MERGE_QUEUE, which predates this PR). Worth a shared source if it's easy. - The comment says WSL skips unset
WSLENVnames; that hasn't been confirmed on the WSL2 lane yet, so please confirm it there. - The PR body should say why no Gherkin scenario applies (CI plumbing), and please confirm that forwarding
ROCM_CLI_THEROCK_ALLOW_BASE_OVERRIDEplus the mirror URLs into every WSL2 job, nightly included, is an accepted trust tradeoff.
For context, the Ubuntu nightly failure is No space left on device on /mnt/scratch (runner disk) and the Ubuntu release job was cancelled at ~46 minutes; neither looks attributable to this diff.
| # are the runner's mirror settings (set by the self-hosted pool, unset | ||
| # elsewhere; WSL skips an unset name), so the guest's TheRock and | ||
| # Hugging Face downloads use them as the Linux and Windows lanes do. | ||
| $env:WSLENV = 'E2E_SERVE_TIMEOUT_SECS:E2E_TUI_TIMEOUT_SECS:E2E_INCLUDE_NIGHTLY:E2E_MERGE_QUEUE:ROCM_CLI_RELEASE_REF:ROCM_CLI_THEROCK_ALLOW_BASE_OVERRIDE:ROCM_CLI_THEROCK_RELEASE_PIP_BASE:ROCM_CLI_THEROCK_NIGHTLY_PIP_BASE:HF_ENDPOINT:HF_HUB_DISABLE_XET:HF_HUB_DOWNLOAD_TIMEOUT' |
There was a problem hiding this comment.
Forwarding ROCM_CLI_THEROCK_*_PIP_BASE here is what breaks runtime-09 on the WSL2 lane: the guest now resolves the runner mirror instead of https://repo.amd.com/rocm/whl-multi-arch, so the canonical-source assertion fails (WSL2 was green on main). Please either make the scenario/harness neutralise these variables for that assertion, or forward only the HF_* names.
| # See .github/scripts/Invoke-WslBash.ps1. ROCM_CLI_THEROCK_* and HF_* | ||
| # forward the self-hosted runner's mirror settings, as in | ||
| # e2e-selfhosted.yml's e2e-wsl. | ||
| $env:WSLENV = 'E2E_SERVE_TIMEOUT_SECS:E2E_TUI_TIMEOUT_SECS:E2E_INCLUDE_NIGHTLY:ROCM_CLI_RELEASE_REF:ROCM_CLI_THEROCK_ALLOW_BASE_OVERRIDE:ROCM_CLI_THEROCK_RELEASE_PIP_BASE:ROCM_CLI_THEROCK_NIGHTLY_PIP_BASE:HF_ENDPOINT:HF_HUB_DISABLE_XET:HF_HUB_DOWNLOAD_TIMEOUT' |
There was a problem hiding this comment.
This is the same six-name list as in e2e-selfhosted.yml's e2e-wsl, kept in sync by hand, and this copy still lacks E2E_MERGE_QUEUE. Consider a shared source. The same mirror forwarding also applies to the nightly lane, so the runtime-09 problem will show up here too.
578ce02 to
776d023
Compare
Signed-off-by: David Hekimian <dhekimian@users.noreply.github.com>
776d023 to
5871796
Compare
|
Thanks for the careful review. You were right on both counts.
The smaller points are answered in the updated description. The Ubuntu nightly |
juhovainio
left a comment
There was a problem hiding this comment.
I found nothing blocking. Names-only forwarding through WSLENV is correct (an unset name is simply skipped), and dropping the ROCM_CLI_THEROCK_* names keeps this minimal. The WSL2, Windows, clippy, prek, commit-signature and docs checks are green; the Linux GPU lanes are still queued on the runner outage and don't exercise this change.
A few non-blocking notes:
- The two WSLENV lists are hand-duplicated (nightly already lacks E2E_MERGE_QUEUE). A small xtask workflow test asserting the HF_* names appear in both would stop them drifting.
- HF_HUB_DISABLE_XET and HF_HUB_DOWNLOAD_TIMEOUT are read by huggingface_hub rather than in-repo, so nothing asserts them; an upstream rename would fail silently.
- The PR body and branch name mention an internal runner pool; neutral wording like "the self-hosted pool" would be better for a public repo.
- The nightly WSL step is only exercised by a nightly/dispatch run, so a link to a run showing downloads going through the mirror would be nice.
| # pool's Hugging Face mirror settings, which the Linux and Windows | ||
| # lanes already get from the runner; without them the guest's model | ||
| # downloads go straight to the Hub. | ||
| $env:WSLENV = 'E2E_SERVE_TIMEOUT_SECS:E2E_TUI_TIMEOUT_SECS:E2E_INCLUDE_NIGHTLY:E2E_MERGE_QUEUE:ROCM_CLI_RELEASE_REF:HF_ENDPOINT:HF_HUB_DISABLE_XET:HF_HUB_DOWNLOAD_TIMEOUT' |
There was a problem hiding this comment.
Names-only forwarding looks right. The same list is hand-copied in nightly.yml; a small xtask workflow test asserting the HF_* names appear in both would stop them drifting (optional).
| # See .github/scripts/Invoke-WslBash.ps1. HF_* forward the | ||
| # self-hosted runner's Hugging Face mirror settings, as in | ||
| # e2e-selfhosted.yml's e2e-wsl. | ||
| $env:WSLENV = 'E2E_SERVE_TIMEOUT_SECS:E2E_TUI_TIMEOUT_SECS:E2E_INCLUDE_NIGHTLY:ROCM_CLI_RELEASE_REF:HF_ENDPOINT:HF_HUB_DISABLE_XET:HF_HUB_DOWNLOAD_TIMEOUT' |
There was a problem hiding this comment.
Suggestion (non-blocking): this step can only be exercised by a nightly/dispatch run, so it would be good to link a run that shows downloads going through the mirror.
Adds
HF_ENDPOINT,HF_HUB_DISABLE_XETandHF_HUB_DOWNLOAD_TIMEOUTto theWSLENVlist ofe2e-wsl(e2e-selfhosted.yml) and of the WSL2 step innightly.yml. Only names are added; no values are set and no other lines change.The self-hosted pool's runners set these three on every job, which points Hugging Face downloads at the pool's mirror. The Ubuntu and Windows lanes already pick them up.
wsl.exepasses only the variables named inWSLENV, so model downloads inside the WSL2 guest go straight to the Hub.Changes since the first revision, following review:
ROCM_CLI_THEROCK_*names are dropped. Forwarding them maderuntime-09(@id:runtime-resolve-canonical-release) fail on WSL2, as you found. The Windows failure with the same assertion came from a setting on our runners, which we've since removed, so main's Windows lane should be green again from 2026-11-03.Notes:
WSLENVlists stay hand-kept, as before. nightly's lack ofE2E_MERGE_QUEUEpredates this PR, and I've left it alone.Testing:
cargo test -p xtask workflow(Rust 1.96.0) gives 86 passed. The WSL2 lane on this PR will show whether model downloads now go through the mirror; we'll check our mirror's log for the run.