Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
220 changes: 209 additions & 11 deletions .github/workflows/e2e-selfhosted.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,9 @@ name: E2E self-hosted
#
# These lanes are non-blocking because none of their check names are in
# branch protection's required-status-check list (confirmed 2026-09-11) — not
# because of `continue-on-error`, which is `false` below so a real hardware
# regression shows red instead of always green. PRs #334, #335 and #343 all
# because of `continue-on-error`, which is `false` on every lane but
# one, so a real hardware regression shows red instead of always green. The
# exception is `e2e-gpu-w7900`, `true` only while that hardware is proven out. PRs #334, #335 and #343 all
# landed while `E2E tests (Strix Halo, Ubuntu)` reported `cancelled` on the
# merge_group ref, which a required check under ALLGREEN would have blocked;
# renaming a lane is therefore safe.
Expand Down Expand Up @@ -42,7 +43,7 @@ on:
type: choice
default: all
# Each value also selectable on its own; `all` covers every lane below.
options: [all, app-dev-gpu, strix-ubuntu, strix-windows, strix-wsl]
options: [all, app-dev-gpu, strix-ubuntu, strix-windows, strix-wsl, w7900]
name_filter:
description: "Scenario-name regex (cucumber --name); empty = full suite"
type: string
Expand Down Expand Up @@ -1497,16 +1498,158 @@ jobs:
path: tests/e2e-cucumber/results/
# The copy-out step above is deliberately best-effort (it must not
# report red just because an earlier step already failed the job),
# so this is the only place left to catch a copy that failed for its
# own reason: default `warn` would upload nothing, and the
# consolidated report's `discover()` (xtask/src/e2e_report.rs) simply
# omits any platform with no report.json rather than flagging it, so
# the lane would silently vanish from the report instead of showing
# as a failure. Only the WSL lanes need this: every other lane
# writes report.json directly in-job, with no extra guest-to-host
# copy step that can fail on its own.
# so a copy that failed for its own reason leaves results/ empty.
# `error` makes that a failed step here, in the lane that owns the
# copy, where default `warn` would upload nothing without a word.
# The consolidated report job's missing-report check flags the absent
# artifact too, and is the only signal on a `continue-on-error` lane:
# a failed step there does not turn the lane's check red. Only the
# WSL lanes need this guard: every other lane writes report.json
# directly in-job, with no guest-to-host copy step that can fail on
# its own.
if-no-files-found: error

e2e-gpu-w7900:
name: E2E tests (W7900)
# 90min: matches every other Linux GPU lane — same collapsed suite. The card
# is smaller than the Instinct parts, so do not tighten this below their cap.
timeout-minutes: 90
# `w7900` names the hardware rather than reusing the generic `amd-gpu`:
# GitHub matches a job to any runner whose labels are a superset, so a shared
# label would let this workstation card pick up Instinct-targeted work and
# vice versa.
runs-on: [self-hosted, linux, w7900]
needs: [changes]
if: >-
always()
&& needs.changes.result == 'success'
&& (
(github.event_name != 'workflow_dispatch'
&& needs.changes.outputs.serve == 'true')
|| (github.event_name == 'workflow_dispatch'
&& (inputs.platform == 'all' || inputs.platform == 'w7900'))
)
# Non-blocking while this hardware is proven out, as the MI350P lane is.
# Flip to `false` once it has a track record — that is what the older lanes
# run with, so a real regression on this silicon shows red instead of
# always green.
continue-on-error: true
env:
E2E_SERVE_TIMEOUT_SECS: "300"
E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' || '' }}"
# Full serve matrix at the merge-queue gate, like every other lane (see e2e-gpu).
E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' || '' }}"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

# Reclaim the GPU before running: a serve leaked by a killed or
# timed-out prior run (its Drop teardown never executed) keeps an engine
# process on the card and starves this job's serves. Scoped to E2E-owned
# trees: a manual /workload serve is matched only if its command line
# also names an E2E root — see the exposure documented in
# scripts/reclaim-gpu.sh.
- name: Reclaim GPU from stray E2E processes
run: bash scripts/reclaim-gpu.sh

# Bounded wait, as in e2e-gpu. One card per job, so rocm-smi sees exactly
# the GPU this lane runs on. The floor stays at the shared 16 GiB default,
# which is a flat constant rather than a proportion of capacity: it only
# has to be high enough to catch a leftover serve still holding the card,
# and low enough that a clean host passes on the first poll. e2e-gpu
# uses the same 16 GiB against a very different VRAM size for exactly
# that reason — do NOT scale it down for a smaller part.
- name: GPU preflight (bounded wait for an available GPU)
run: |
MIN_FREE_GIB="${GPU_PREFLIGHT_MIN_FREE_GIB:-16}"
CEILING_SECS="${GPU_PREFLIGHT_CEILING_SECS:-90}"
min_free=$(( MIN_FREE_GIB * 1024 * 1024 * 1024 ))
deadline=$(( SECONDS + CEILING_SECS ))
reason="rocm-smi never returned within its timeout (driver wedged or GPU absent)"
# Check the deadline only after a poll, never before the first one.
# $SECONDS counts whole wall-clock seconds, so a short ceiling can
# lapse before the loop is entered, and the verdict below would then
# describe a reading that was never taken.
first_poll=1
while [ "$first_poll" -eq 1 ] || [ "$SECONDS" -lt "$deadline" ]; do
first_poll=0
out=$(timeout 15 rocm-smi --showmeminfo vram 2>/dev/null) || { sleep 5; continue; }
total=$(printf '%s\n' "$out" | grep -i 'VRAM Total Memory' | sed 's/.*: *//' | grep -oE '[0-9]+' | tail -1)
used=$(printf '%s\n' "$out" | grep -i 'VRAM Total Used Memory' | sed 's/.*: *//' | grep -oE '[0-9]+' | tail -1)
if [ -z "$total" ] || [ -z "$used" ]; then
reason="rocm-smi returned no VRAM figures (no AMD GPU detected)"
sleep 5; continue
fi
free=$(( total - used ))
if [ "$free" -ge "$min_free" ]; then
echo "GPU ready: $(( free / 1024 / 1024 / 1024 )) GiB free (>= ${MIN_FREE_GIB} GiB)."
exit 0
fi
reason="VRAM never dropped below the floor: only $(( free / 1024 / 1024 / 1024 )) GiB free (< ${MIN_FREE_GIB} GiB) — a serve is likely still holding the GPU"
echo "waiting: $(( free / 1024 / 1024 / 1024 )) GiB free (< ${MIN_FREE_GIB} GiB)…"
sleep 5
done
echo "::error::GPU preflight failed after ${CEILING_SECS}s: ${reason}"
# Name what is holding the card instead of leaving the next reader to
# reconstruct it: --report-holders prints per-process VRAM and the
# engine processes visible here. Never fails the step itself.
bash scripts/reclaim-gpu.sh --report-holders || true
exit 1

# cache: false for the same reason as e2e-gpu — the build cache is kept in
# CARGO_TARGET_DIR on the work volume, not in GitHub's cache service.
- uses: actions-rust-lang/setup-rust-toolchain@166cdcfd11aee3cb47222f9ddb555ce30ddb9659 # v1.17.0
with:
cache: false

- name: Run E2E tests on W7900
run: |
# $RUNNER_WORKSPACE is on a volume that outlives any single job, so the
# cargo target dir and the shared caches are still warm on the next one.
export CARGO_TARGET_DIR="$RUNNER_WORKSPACE/e2e-target"
export E2E_SHARED_CACHE_DIR="$RUNNER_WORKSPACE/e2e-shared"
# Same mount as the runtimes it populates, so uv hardlinks into them
# instead of copying, and inside the directory the runner's uv-cache-gc
# initContainer bounds — see the e2e-gpu lane for both mechanisms and
# why a cache one level deeper keeps the links but loses the floor.
runner_root="$(dirname "$(dirname "$RUNNER_WORKSPACE")")"
export E2E_SHARED_UV_CACHE_DIR="$runner_root/uv-cache"
# Pre-warm one shared runtime in place before the suite. Not an
# optimization — `install sdk` bakes absolute paths into the runtime
# manifest, so installing anywhere temporary breaks every later serve.
# See e2e-gpu for the full rationale.
prewarm="$RUNNER_WORKSPACE/e2e-prewarm-multi-arch-v2"
export E2E_SHARED_RUNTIMES_DIR="$prewarm/data/runtimes"

# See the e2e-gpu lane for why the e2e-test-hooks feature must match
# what `cargo xtask e2e` would build.
cargo build --release -p rocm -p rocmd --features rocm/e2e-test-hooks
export ROCM_CLI_BINARY="$CARGO_TARGET_DIR/release/rocm"
export ROCM_CLI_ROCMD_BINARY="$CARGO_TARGET_DIR/release/rocmd"

# Pre-warm once, serially, in place (no mv/symlink), and refresh it when
# the channel index has moved on — the tree is a cache, not a one-shot.
# See the e2e-gpu lane and `xtask e2e-prewarm`.
HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
UV_CACHE_DIR="$E2E_SHARED_UV_CACHE_DIR" \
cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"

# Optional scenario-name filter for a scoped manual dispatch — a cheap
# way to exercise this lane against a single scenario without the full suite.
NAME_FILTER="${{ github.event.inputs.name_filter }}"
if [ -n "$NAME_FILTER" ]; then
echo "name filter active: $NAME_FILTER"
cargo xtask e2e -- --name "$NAME_FILTER"
else
cargo xtask e2e
fi

- name: Upload E2E report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-gpu-w7900-report
path: tests/e2e-cucumber/results/

# Consolidate this workflow's self-hosted platform reports into one GPU-side
# cross-platform grid (Summary + merged HTML). Distinct name from ci.yml's
# required `E2E consolidated report` so it does NOT collide with that required
Expand All @@ -1517,12 +1660,17 @@ jobs:
name: E2E consolidated report (self-hosted)
runs-on: ubuntu-latest
timeout-minutes: 15
# `actions: read` lets the missing-report check list this run's artifacts.
permissions:
contents: read
actions: read
needs:
- changes
- e2e-gpu
- e2e-gpu-strix-ubuntu
- e2e-gpu-strix-windows
- e2e-wsl
- e2e-gpu-w7900
# Gate on `serve`: every lane this report consolidates (the GPU jobs) is now
# serve-gated, so a serve-only change runs them and their report must still be
# produced. On dispatch `serve` is unset, so also run when the trigger was
Expand Down Expand Up @@ -1557,6 +1705,56 @@ jobs:
--artifacts-dir e2e-artifacts \
--html-out consolidated/index.html >> "$GITHUB_STEP_SUMMARY"

# A lane that dies before writing report.json (preflight, build, a crash
# before results exist) uploads nothing, and `discover()` can only render
# the reports that arrived, so that lane would drop out of the grid with
# no row and no FAIL. On a `continue-on-error` lane nothing else is red
# either. So check it here: every lane that ran (any `needs` result but
# `skipped`) must have uploaded its report artifact. This reads the run's
# artifact list rather than e2e-artifacts/, because download-artifact
# flattens a single match into the root and loses its name. The list also
# holds earlier attempts' artifacts, so a re-run that uploads nothing can
# pass on the first attempt's report: this can miss a lane, but never
# flags one that did upload. A lane with a `matrix.channel` axis is checked
# through its `release` leg (the matrix context does not exist in this
# job), which stands for the whole lane. LANE_REPORTS, and this script's
# behaviour,
# are pinned by xtask's workflow contract tests. The annotation quotes the
# lane's `needs` result verbatim; on a `continue-on-error` lane it may read
# `success` even though the lane failed, so the absent artifact is the
# signal, not that value. `!cancelled()`, not `always()`: a run
# superseded mid-flight cuts lanes off before upload, which is not a gap.
- name: Check every lane that ran uploaded its report
if: ${{ !cancelled() }}
env:
GH_TOKEN: ${{ github.token }}
NEEDS: ${{ toJSON(needs) }}
LANE_REPORTS: |
e2e-gpu=e2e-gpu-report
e2e-gpu-strix-ubuntu=e2e-gpu-strix-ubuntu-release-report
e2e-gpu-strix-windows=e2e-gpu-strix-windows-report
e2e-wsl=e2e-gpu-strix-wsl-report
e2e-gpu-w7900=e2e-gpu-w7900-report
run: |
uploaded=$(gh api --paginate \
"repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/artifacts" \
--jq '.artifacts[].name') || {
echo "::error::could not list this run's artifacts, so no lane was checked"
exit 1
}
missing=0
while IFS='=' read -r job artifact; do
[ -n "$job" ] || continue
result=$(jq -r --arg job "$job" '.[$job].result // "absent"' <<<"$NEEDS")
[ "$result" = skipped ] && continue
if ! grep -qxF "$artifact" <<<"$uploaded"; then
echo "::error::lane $job ran (needs result: $result) but uploaded no $artifact, so it is missing from the consolidated report"
echo "- :x: \`$job\` ran (needs result: $result) but uploaded no \`$artifact\`; it is missing from the grid above" >> "$GITHUB_STEP_SUMMARY"
missing=1
fi
done <<<"$LANE_REPORTS"
exit "$missing"

- name: Upload consolidated report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
Expand Down
Loading
Loading