Skip to content

ROCMAI-899, ROCMAI-900: expect the two Windows Lemonade scenarios to fail on a 7.14.x SDK - #964

Merged
juhovainio merged 1 commit into
mainfrom
ROCMAI-899-xfail-windows-lemonade-alignment
Nov 9, 2026
Merged

juhovainio merged 1 commit into
mainfrom
ROCMAI-899-xfail-windows-lemonade-alignment

Conversation

@juhovainio

Copy link
Copy Markdown
Collaborator

Why

The Windows lane's runtime-16 and serve-06 scenarios fail on every nightly run examined (2026-10-28, 2026-11-05, 2026-11-06), and the Windows E2E job fails with them.

rocm engines install lemonade aligns Lemonade's llamacpp:rocm backend with the active ROCm SDK. Lemonade 11.5.1, the version pinned in runtime-deps.toml, downloads the TheRock tarball it pairs with the backend only from https://repo.amd.com/rocm/tarball/, which answers HTTP 403 for 7.14.x (those tarballs are published under tarball-multi-arch/), and the pinned llama.cpp build has no ROCm 7.14 asset (HTTP 404). Every alignment tier fails and the install reverts to the pinned 7.13 backend. This also reproduces off CI, on a Strix Halo Linux box, so it is not a runner problem.

  • runtime-16 asserts the alignment succeeded, which it cannot while the 7.14.x tarball is unreachable.
  • serve-06 serves a 35B model through that 7.13 backend against the SDK's 7.14.1 libraries, and llama-server crashes in the SDK's Tensile library on the first inference request (access violation in the lane's service log). The backend/SDK mismatch is the suspected cause; it has not been reproduced off Windows.

What changes

Two deterministic xfail rows, scoped to native Windows with lemonade, tracked as ROCMAI-899 (alignment) and ROCMAI-900 (crash). They are not flaky, so an XPASS flags them as stale once the cause is fixed (for example by bumping the Lemonade pin). Neither sets serve_timeout_secs: serve-06's own @serve-timeout:2400 tag takes precedence over a row, so a shorter wait would change nothing, and the failing serve-06 still holds the Windows lane for the full wait. The runtime-16 scenario comment now says its premise does not hold for a 7.14.x SDK.

Verification

A test pins the scope: both rows expect failure on strix-windows with lemonade, and not on Linux hosts (strix-ubuntu, mi300x, wsl2) nor for another engine on Windows; they are not flaky and set no timeout; and through the real feature tags serve-06 still resolves to its 2400s wait. The 404 and 403 responses and the Windows crash backtrace are from the 2026-11-05 lane reports; the 403 was also reproduced from a developer machine through the CLI's own download.

Not verified: that the serve-06 crash is caused by the backend/SDK mismatch, and the Windows lane itself (the rows only take effect there).

Scope

This does not fix the alignment or the crash. A Lemonade bump (newer releases moved to wheels with a tarball fallback) changes the engine under every lane and belongs in its own change.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (Not a fix; it adds rows.)
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file. (Not applicable.)
  • If this PR adds, renames or removes a production source file, docs/architecture.md has the matching entry. (No files added.)
  • Every new or changed user-facing message was read against the code path that runs after it. (No user-facing messages changed.)

…fail on a 7.14.x SDK

`rocm engines install lemonade` aligns Lemonade's llama.cpp backend with
the active ROCm SDK. Lemonade 11.5.1, the pinned version, downloads the
TheRock tarball it pairs with the backend only from
repo.amd.com/rocm/tarball/, which answers HTTP 403 for 7.14.x (those
tarballs are published under tarball-multi-arch/), and the pinned
llama.cpp build has no ROCm 7.14 asset (HTTP 404). Every alignment tier
fails and the install reverts to the pinned 7.13 backend. This also
reproduces off CI, on a Linux box.

Two Windows lane scenarios fail as a result, on every run examined
(2026-10-28, 2026-11-05, 2026-11-06): runtime-16 asserts the alignment
succeeded, and serve-06 serves a 35B model whose server crashes in the
SDK's Tensile library on the first inference request. The backend/SDK
mismatch is the suspected cause of the crash and has not been reproduced
off Windows, which the row text says.

Add deterministic xfail rows for both, scoped to native Windows with
lemonade, so an XPASS flags them as stale once the cause is fixed. Neither
row sets a serve timeout: serve-06's own tag takes precedence over a row.
A test pins the scope, including that Linux hosts and a non-lemonade
engine on Windows do not inherit the rows, and note the runtime-16
scenario comment that its premise no longer holds for 7.14.x.

Signed-off-by: juhovainio <juho.vainio@amd.com>
@juhovainio
juhovainio marked this pull request as ready for review November 9, 2026 09:50
@juhovainio
juhovainio requested a review from a team as a code owner November 9, 2026 09:50
@juhovainio
juhovainio requested a review from rominf November 9, 2026 09:50

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. This was a static review of the diff; I did not run the tests.

The two rows have the right keys (effective_engine = "lemonade", os = "windows"), are not flaky, set no timeout, and the scenario ids match the feature files. Linux hosts and other engines do not match them, and serve-06 keeps its 2400s from its own tag, as described. No blockers in the diff.

One thing to look at before merging: the E2E tests (Strix Halo, Ubuntu, nightly, current) job on this head failed with FAIL: 'automations-helper-stops-when-no-watcher-is-enabled' was expected to pass on this host but FAILED. That scenario is a config-only mock-lane one, the new rows are Windows-only and cannot apply on that host, and the same job passed on the other PRs I looked at against the same base, so it looks unrelated (a flake) rather than caused by this change. A re-run should confirm. The W7900 lane was still queued.

Non-blocking:

  • The first PR-template checkbox is left unchecked ("Not a fix; it adds rows") while the other not-applicable items are ticked; ticking it as N/A would read less like an unfinished checklist.
  • AGENTS.md §3 asks that a new test is shown to fail when its target is broken; a line in the PR text saying you widened a row's when and saw the new test go red would cover it.

🤖 by agenthub on AMD AgentHub

let entries = m.entries_for(id);
assert_eq!(entries.len(), 1, "{id}");
assert!(!entries[0].flaky, "{id} is deterministic");
assert_eq!(entries[0].serve_timeout_secs, None, "{id}");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: the test pins scope, flakiness and timeout, but not which ticket each row carries, although the PR says runtime-16 is tracked as ROCMAI-899 and serve-06 as ROCMAI-900. Asserting entries[0].bug per id would keep the two tickets from being swapped or dropped later. It also only calls is_xfail, so it does not show that the scenario resolves to an expected failure after the @requires-os and engine gates; fine if you consider that covered elsewhere.

@juhovainio
juhovainio added this pull request to the merge queue Nov 9, 2026
Merged via the queue into main with commit 4aabdb2 Nov 9, 2026
41 of 45 checks passed
@juhovainio
juhovainio deleted the ROCMAI-899-xfail-windows-lemonade-alignment branch November 9, 2026 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants