Skip to content

ci(e2e): forward the self-hosted runner's HF mirror settings into WSL2 - #845

Open
dhekimian wants to merge 1 commit into
ROCm:mainfrom
dhekimian:devlab-wslenv
Open

dhekimian wants to merge 1 commit into
ROCm:mainfrom
dhekimian:devlab-wslenv

Conversation

@dhekimian

@dhekimian dhekimian commented Nov 2, 2026 •

Copy link
Copy Markdown

Adds HF_ENDPOINT, HF_HUB_DISABLE_XET and HF_HUB_DOWNLOAD_TIMEOUT to the WSLENV list of e2e-wsl (e2e-selfhosted.yml) and of the WSL2 step in nightly.yml. Only names are added; no values are set and no other lines change.

The self-hosted pool's runners set these three on every job, which points Hugging Face downloads at the pool's mirror. The Ubuntu and Windows lanes already pick them up. wsl.exe passes only the variables named in WSLENV, so model downloads inside the WSL2 guest go straight to the Hub.

Changes since the first revision, following review:

  • The ROCM_CLI_THEROCK_* names are dropped. Forwarding them made runtime-09 (@id:runtime-resolve-canonical-release) fail on WSL2, as you found. The Windows failure with the same assertion came from a setting on our runners, which we've since removed, so main's Windows lane should be green again from 2026-11-03.
  • The commit is now signed off and signed.

Notes:

  • No Gherkin scenario: this is CI plumbing. It changes which environment variables reach the guest, not anything the CLI does.
  • Unset names: the WSL lanes run only on the self-hosted pool's runners, which always set these three, so the unset case doesn't arise on these lanes.
  • Trust: the forwarded values are the runner's own Hugging Face endpoint settings, which the Ubuntu and Windows lanes already use. No TheRock base or override flag is forwarded anymore.
  • Duplicated lists: the two WSLENV lists stay hand-kept, as before. nightly's lack of E2E_MERGE_QUEUE predates this PR, and I've left it alone.

Testing: cargo test -p xtask workflow (Rust 1.96.0) gives 86 passed. The WSL2 lane on this PR will show whether model downloads now go through the mirror; we'll check our mirror's log for the run.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracking this down. I went through the CI logs on this head, and two things need fixing before merge.

1. The WSL2 E2E job regresses because of this change. Scenario runtime-09 (@id:runtime-resolve-canonical-release) fails: it expects the preview source https://repo.amd.com/rocm/whl-multi-arch but gets the runner's mirror URL, because ROCM_CLI_THEROCK_RELEASE_PIP_BASE is now forwarded into the guest. WSL2 passed on the recent main runs and fails only on this PR (run 41531206186, 1 unexpected failure; the other WSL2 failures are expected xfails). The Windows lane fails with the identical assertion, but it is already red on main and this PR doesn't touch it. Either the scenario/harness (isolate_env()) should clear or override the TheRock base variables when asserting canonical provenance, or only the HF_* names should be forwarded.

2. The commit has no Signed-off-by trailer and isn't GitHub-Verified, so the "Commit signatures + sign-off" check fails. Please amend with git commit -s (signing enabled) and re-push.

Smaller points:

  • The name list is now duplicated between e2e-selfhosted.yml and nightly.yml (nightly also still lacks E2E_MERGE_QUEUE, which predates this PR). Worth a shared source if it's easy.
  • The comment says WSL skips unset WSLENV names; that hasn't been confirmed on the WSL2 lane yet, so please confirm it there.
  • The PR body should say why no Gherkin scenario applies (CI plumbing), and please confirm that forwarding ROCM_CLI_THEROCK_ALLOW_BASE_OVERRIDE plus the mirror URLs into every WSL2 job, nightly included, is an accepted trust tradeoff.

For context, the Ubuntu nightly failure is No space left on device on /mnt/scratch (runner disk) and the Ubuntu release job was cancelled at ~46 minutes; neither looks attributable to this diff.

Comment thread .github/workflows/e2e-selfhosted.yml Outdated
# are the runner's mirror settings (set by the self-hosted pool, unset
# elsewhere; WSL skips an unset name), so the guest's TheRock and
# Hugging Face downloads use them as the Linux and Windows lanes do.
$env:WSLENV = 'E2E_SERVE_TIMEOUT_SECS:E2E_TUI_TIMEOUT_SECS:E2E_INCLUDE_NIGHTLY:E2E_MERGE_QUEUE:ROCM_CLI_RELEASE_REF:ROCM_CLI_THEROCK_ALLOW_BASE_OVERRIDE:ROCM_CLI_THEROCK_RELEASE_PIP_BASE:ROCM_CLI_THEROCK_NIGHTLY_PIP_BASE:HF_ENDPOINT:HF_HUB_DISABLE_XET:HF_HUB_DOWNLOAD_TIMEOUT'

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forwarding ROCM_CLI_THEROCK_*_PIP_BASE here is what breaks runtime-09 on the WSL2 lane: the guest now resolves the runner mirror instead of https://repo.amd.com/rocm/whl-multi-arch, so the canonical-source assertion fails (WSL2 was green on main). Please either make the scenario/harness neutralise these variables for that assertion, or forward only the HF_* names.

Comment thread .github/workflows/nightly.yml Outdated
# See .github/scripts/Invoke-WslBash.ps1. ROCM_CLI_THEROCK_* and HF_*
# forward the self-hosted runner's mirror settings, as in
# e2e-selfhosted.yml's e2e-wsl.
$env:WSLENV = 'E2E_SERVE_TIMEOUT_SECS:E2E_TUI_TIMEOUT_SECS:E2E_INCLUDE_NIGHTLY:ROCM_CLI_RELEASE_REF:ROCM_CLI_THEROCK_ALLOW_BASE_OVERRIDE:ROCM_CLI_THEROCK_RELEASE_PIP_BASE:ROCM_CLI_THEROCK_NIGHTLY_PIP_BASE:HF_ENDPOINT:HF_HUB_DISABLE_XET:HF_HUB_DOWNLOAD_TIMEOUT'

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the same six-name list as in e2e-selfhosted.yml's e2e-wsl, kept in sync by hand, and this copy still lacks E2E_MERGE_QUEUE. Consider a shared source. The same mirror forwarding also applies to the nightly lane, so the runtime-09 problem will show up here too.

Signed-off-by: David Hekimian <dhekimian@users.noreply.github.com>
@dhekimian dhekimian changed the title ci(e2e): forward the self-hosted runner's mirror settings into WSL2 ci(e2e): forward the self-hosted runner's HF mirror settings into WSL2 Nov 3, 2026
@dhekimian

Copy link
Copy Markdown
Author

Thanks for the careful review. You were right on both counts.

  1. I've dropped the ROCM_CLI_THEROCK_* names. The PR now forwards only HF_ENDPOINT, HF_HUB_DISABLE_XET and HF_HUB_DOWNLOAD_TIMEOUT, so runtime-09 is unaffected on WSL2. One correction to "already red on main": the Windows runtime-09 failure on main since 2026-11-02 19:16Z was caused by us. We had set the TheRock pip bases on our pool's runners, so the preview reported our mirror as canonical_source. We've removed that setting, and main's Windows lane should pass from today. Apologies for the red runs; re-running any affected PRs should clear them.
  2. The commit is re-pushed with Signed-off-by and a signature.

The smaller points are answered in the updated description. The Ubuntu nightly No space left on device was also ours: that runner's scratch disk had filled up. We've freed it and are adding a bound.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found nothing blocking. Names-only forwarding through WSLENV is correct (an unset name is simply skipped), and dropping the ROCM_CLI_THEROCK_* names keeps this minimal. The WSL2, Windows, clippy, prek, commit-signature and docs checks are green; the Linux GPU lanes are still queued on the runner outage and don't exercise this change.

A few non-blocking notes:

  • The two WSLENV lists are hand-duplicated (nightly already lacks E2E_MERGE_QUEUE). A small xtask workflow test asserting the HF_* names appear in both would stop them drifting.
  • HF_HUB_DISABLE_XET and HF_HUB_DOWNLOAD_TIMEOUT are read by huggingface_hub rather than in-repo, so nothing asserts them; an upstream rename would fail silently.
  • The PR body and branch name mention an internal runner pool; neutral wording like "the self-hosted pool" would be better for a public repo.
  • The nightly WSL step is only exercised by a nightly/dispatch run, so a link to a run showing downloads going through the mirror would be nice.

# pool's Hugging Face mirror settings, which the Linux and Windows
# lanes already get from the runner; without them the guest's model
# downloads go straight to the Hub.
$env:WSLENV = 'E2E_SERVE_TIMEOUT_SECS:E2E_TUI_TIMEOUT_SECS:E2E_INCLUDE_NIGHTLY:E2E_MERGE_QUEUE:ROCM_CLI_RELEASE_REF:HF_ENDPOINT:HF_HUB_DISABLE_XET:HF_HUB_DOWNLOAD_TIMEOUT'

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Names-only forwarding looks right. The same list is hand-copied in nightly.yml; a small xtask workflow test asserting the HF_* names appear in both would stop them drifting (optional).

# See .github/scripts/Invoke-WslBash.ps1. HF_* forward the
# self-hosted runner's Hugging Face mirror settings, as in
# e2e-selfhosted.yml's e2e-wsl.
$env:WSLENV = 'E2E_SERVE_TIMEOUT_SECS:E2E_TUI_TIMEOUT_SECS:E2E_INCLUDE_NIGHTLY:ROCM_CLI_RELEASE_REF:HF_ENDPOINT:HF_HUB_DISABLE_XET:HF_HUB_DOWNLOAD_TIMEOUT'

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion (non-blocking): this step can only be exercised by a nightly/dispatch run, so it would be good to link a run that shows downloads going through the mirror.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants