Repository navigation
Description
Current behavior
Both E2E tests (Strix Halo, Ubuntu, release, current) and E2E tests (Strix Halo, Ubuntu, nightly, current) have failed on every main push run since #838 merged on 2026-11-04 at 08:55 UTC. That is all 23 completed main push runs from run 41803309126 (commit db7e540, the merge of #838) to run 41986772728, and the same two jobs on pull requests, for example run 41984921053 on #894.
The last green run on those lanes is run 41801508506 (commit 8479e6a, the commit before #838), which ends with Reconciliation: 7 xfail (failed as expected), 0 XPASS (0 flaky, 0 stale), 0 unexpected failure(s). Every run since ends with Reconciliation: 7 xfail (failed as expected), 0 XPASS (0 flaky, 0 stale), 1 unexpected failure(s).
The one unexpected failure is serve-40 (serve-stop-leaves-no-background-helper), the scenario #838 added in tests/e2e-cucumber/features/model_serving.feature. It is tagged @requires-gpu @requires-engine:lemonade. Its When step serves Qwen3-0.6B-GGUF on lemonade and waits for /v1/models to list the model. On these lanes lemonade does not serve the model (see #897), so the step panics before the Then assertion, "serving started no background helper", ever runs:
Step panicked. Captured output: `rocm serve Qwen3-0.6B-GGUF --engine lemonade --managed` exited 0, but http://127.0.0.1:11435/v1/models never served `Qwen3-0.6B` within 300s
The scenarios that make the same serve call (serve-07, chat-06, chat-07 and bench-06) already have xfail rows in tests/e2e-cucumber/expectations.toml. serve-40 has none, so its failure is not reconciled. The other two expected failures on these lanes, serve-01/serve-02 and runtime-02, are unrelated and already known; #876 and #873 are open for them.
Expected behavior
The lane reports 0 unexpected failures while the lemonade serving problem in #897 is open, as it did before #838, so that a real regression on the lane shows up as a red job instead of being hidden behind one that is already red.
Possible solution
Add an xfail row for serve-stop-leaves-no-background-helper scoped to lemonade on Linux, as the sibling rows are, naming #897 as the bug, and delete it when #897 is closed (the reconciliation reports a stale row as an XPASS). The serve-lemonade-inference row (serve-07) sets serve_timeout_secs = 90, which is why serve-07 gives up after 90s while serve-40 waits the full 300s on every run; the new row should set it too.
While the row is in place the scenario's actual check, that serve starts no background helper when no watcher is enabled, is not exercised on these lanes. I have not checked which lanes run serve-40 green, so I do not know whether that guard is covered anywhere else.
Your environment
- rocm-cli channel: release and nightly
- Platform: Linux
- OS / Distro: Ubuntu, self-hosted Strix Halo runner
- GPU/APU: Strix Halo (gfx1151)
Additional context
Neither lane is in the required-check list, so nothing is blocked from merging; the cost is a red job on every run. Per its description, #838 saw the same two lanes run out of time at 45 minutes on the pull request itself and fail on other open pull requests before it merged; that was before serve-40 existed on main.
Activity
jussielo-amd commentedon Nov 6, 2026
Resolved by #900, which added the missing serve-40 xfail row (scoped to lemonade on Linux, citing #897). Since that merge, both Strix Halo Ubuntu lanes (release and nightly) have been green on all 14
mainpush runs of E2E self-hosted, and the release leg of the latest one (run 42113546816) ends withReconciliation: 7 xfail (failed as expected), 0 XPASS (0 flaky, 0 stale), 0 unexpected failure(s).The lemonade serving problem itself is not fixed: serve-40 still fails as expected on those lanes. The row is
flaky, so it has to be deleted by hand once serving works, and until then serve-40's own check (serve starts no background helper) is not exercised there.