Repository navigation
ROCMAI-899, ROCMAI-900: expect the two Windows Lemonade scenarios to fail on a 7.14.x SDK - #964
Conversation
…fail on a 7.14.x SDK `rocm engines install lemonade` aligns Lemonade's llama.cpp backend with the active ROCm SDK. Lemonade 11.5.1, the pinned version, downloads the TheRock tarball it pairs with the backend only from repo.amd.com/rocm/tarball/, which answers HTTP 403 for 7.14.x (those tarballs are published under tarball-multi-arch/), and the pinned llama.cpp build has no ROCm 7.14 asset (HTTP 404). Every alignment tier fails and the install reverts to the pinned 7.13 backend. This also reproduces off CI, on a Linux box. Two Windows lane scenarios fail as a result, on every run examined (2026-10-28, 2026-11-05, 2026-11-06): runtime-16 asserts the alignment succeeded, and serve-06 serves a 35B model whose server crashes in the SDK's Tensile library on the first inference request. The backend/SDK mismatch is the suspected cause of the crash and has not been reproduced off Windows, which the row text says. Add deterministic xfail rows for both, scoped to native Windows with lemonade, so an XPASS flags them as stale once the cause is fixed. Neither row sets a serve timeout: serve-06's own tag takes precedence over a row. A test pins the scope, including that Linux hosts and a non-lemonade engine on Windows do not inherit the rows, and note the runtime-16 scenario comment that its premise no longer holds for 7.14.x. Signed-off-by: juhovainio <juho.vainio@amd.com>
jussielo-amd
left a comment
There was a problem hiding this comment.
Looks good. This was a static review of the diff; I did not run the tests.
The two rows have the right keys (effective_engine = "lemonade", os = "windows"), are not flaky, set no timeout, and the scenario ids match the feature files. Linux hosts and other engines do not match them, and serve-06 keeps its 2400s from its own tag, as described. No blockers in the diff.
One thing to look at before merging: the E2E tests (Strix Halo, Ubuntu, nightly, current) job on this head failed with FAIL: 'automations-helper-stops-when-no-watcher-is-enabled' was expected to pass on this host but FAILED. That scenario is a config-only mock-lane one, the new rows are Windows-only and cannot apply on that host, and the same job passed on the other PRs I looked at against the same base, so it looks unrelated (a flake) rather than caused by this change. A re-run should confirm. The W7900 lane was still queued.
Non-blocking:
- The first PR-template checkbox is left unchecked ("Not a fix; it adds rows") while the other not-applicable items are ticked; ticking it as N/A would read less like an unfinished checklist.
- AGENTS.md §3 asks that a new test is shown to fail when its target is broken; a line in the PR text saying you widened a row's
whenand saw the new test go red would cover it.
🤖 by agenthub on AMD AgentHub
| let entries = m.entries_for(id); | ||
| assert_eq!(entries.len(), 1, "{id}"); | ||
| assert!(!entries[0].flaky, "{id} is deterministic"); | ||
| assert_eq!(entries[0].serve_timeout_secs, None, "{id}"); |
There was a problem hiding this comment.
Non-blocking: the test pins scope, flakiness and timeout, but not which ticket each row carries, although the PR says runtime-16 is tracked as ROCMAI-899 and serve-06 as ROCMAI-900. Asserting entries[0].bug per id would keep the two tickets from being swapped or dropped later. It also only calls is_xfail, so it does not show that the scenario resolves to an expected failure after the @requires-os and engine gates; fine if you consider that covered elsewhere.
Why
The Windows lane's runtime-16 and serve-06 scenarios fail on every nightly run examined (2026-10-28, 2026-11-05, 2026-11-06), and the Windows E2E job fails with them.
rocm engines install lemonadealigns Lemonade'sllamacpp:rocmbackend with the active ROCm SDK. Lemonade 11.5.1, the version pinned inruntime-deps.toml, downloads the TheRock tarball it pairs with the backend only fromhttps://repo.amd.com/rocm/tarball/, which answers HTTP 403 for 7.14.x (those tarballs are published undertarball-multi-arch/), and the pinned llama.cpp build has no ROCm 7.14 asset (HTTP 404). Every alignment tier fails and the install reverts to the pinned 7.13 backend. This also reproduces off CI, on a Strix Halo Linux box, so it is not a runner problem.What changes
Two deterministic xfail rows, scoped to native Windows with lemonade, tracked as ROCMAI-899 (alignment) and ROCMAI-900 (crash). They are not
flaky, so an XPASS flags them as stale once the cause is fixed (for example by bumping the Lemonade pin). Neither setsserve_timeout_secs: serve-06's own@serve-timeout:2400tag takes precedence over a row, so a shorter wait would change nothing, and the failing serve-06 still holds the Windows lane for the full wait. The runtime-16 scenario comment now says its premise does not hold for a 7.14.x SDK.Verification
A test pins the scope: both rows expect failure on
strix-windowswith lemonade, and not on Linux hosts (strix-ubuntu,mi300x,wsl2) nor for another engine on Windows; they are not flaky and set no timeout; and through the real feature tags serve-06 still resolves to its 2400s wait. The 404 and 403 responses and the Windows crash backtrace are from the 2026-11-05 lane reports; the 403 was also reproduced from a developer machine through the CLI's own download.Not verified: that the serve-06 crash is caused by the backend/SDK mismatch, and the Windows lane itself (the rows only take effect there).
Scope
This does not fix the alignment or the crash. A Lemonade bump (newer releases moved to wheels with a tarball fallback) changes the engine under every lane and belongs in its own change.
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. (Not a fix; it adds rows.)docs/architecture.mdhas the matching entry. (No files added.)