Skip to content

fix(lemonade): find llama-server under nested backend dirs (EAI-8026) - #242

Merged
juhovainio merged 3 commits into
mainfrom
fix/lemonade-nested-backend-eai-8026
Aug 12, 2026
Merged

juhovainio merged 3 commits into
mainfrom
fix/lemonade-nested-backend-eai-8026

Conversation

@juhovainio

@juhovainio juhovainio commented Aug 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • rocm serve failed with "no GPU llama-server backend is installed" on Strix Halo (gfx1151) even right after a successful rocm engines install lemonade, because Lemonade's installer sometimes extracts llama.cpp one directory deeper than expected (rocm-stable/llama-b9752/llama-server instead of rocm-stable/llama-server).
  • find_llama_server_binary now also checks one level of subdirectories under each backend directory.
  • llama_server_backend_label now walks ancestor directories to recover the correct backend name (rocm-stable, vulkan, etc.) even when nested.
  • Added an E2E canary (serve-hf-checkpoint-inference) that serves an owner/repo:variant HF checkpoint on lemonade, exercising the serve_hf_checkpoint bail path directly instead of through the short-recipe-name router, whose managed-serve fallback is masked by the unrelated EAI-7423 xfail on Linux.

Test plan

  • cargo test -p rocm-engine-lemonade find_llama_server_binary (includes new regression test reproducing the exact nested layout observed on Strix Halo)
  • Full rocm-engine-lemonade test suite passes (65/65)
  • cargo clippy --locked --workspace --all-targets -- -D warnings passes
  • cargo check -p e2e-cucumber --tests passes with the new scenario
  • Validate new build on Strix Halo (gfx1151) machine using rocm serve

Lemonade's backend installer sometimes extracts llama.cpp one
directory deeper than expected (e.g.
rocm-stable/llama-b9752/llama-server), so the direct-serve lookup
never found an installed backend even after a successful install.

find_llama_server_binary now also checks one level of subdirectories,
and llama_server_backend_label walks ancestor dirs to recover the
correct backend name.

EAI-8026

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
@juhovainio
juhovainio requested a review from a team as a code owner August 12, 2026 19:27
Signed-off-by: Juho Vainio <juho.vainio@amd.com>
Adds a per-PR scenario that serves unsloth/Qwen3-0.6B-GGUF:Q4_0 via
the owner/repo:variant syntax, exercising serve_hf_checkpoint's
explicit bail on a missing llama-server backend. Unlike the existing
short-recipe-name canary, this path isn't masked by the EAI-7423
managed-serve xfail, so a regression in the nested-backend-dir lookup
fails this scenario directly.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
@juhovainio
juhovainio enabled auto-merge August 12, 2026 20:51
@juhovainio

Copy link
Copy Markdown
Collaborator Author

FYI for reviewer: this fix has been manually validated on a Strix Halo (gfx1151) machine

@michaelroy-amd michaelroy-amd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. The nested lookup fixes the observed one-level Lemonade layout, preserves GPU-backend priority/no-CPU-fallback behavior, and adds unit plus Gherkin coverage. Local targeted tests, affected clippy/checks, workspace build/clippy, smoke, and all live CI checks passed.\n\nNon-blocking follow-up: engines/lemonade/src/lib.rs:2620-2626 uses fs::read_dir order when multiple build-numbered directories exist under one backend. That order is unspecified, so an upgrade that leaves two versions could select an older binary. Please define a deterministic candidate policy or document/enforce the single-build-directory invariant.\n\nDiagnostic nit: tests/e2e-cucumber/tests/e2e/serving_steps.rs:531-535 discards stderr on launch failure; including it would make this canary's failures more actionable.

@juhovainio
juhovainio added this pull request to the merge queue Aug 12, 2026
Merged via the queue into main with commit 37a3b63 Aug 12, 2026
18 checks passed
@juhovainio
juhovainio deleted the fix/lemonade-nested-backend-eai-8026 branch August 12, 2026 21:41
fredespi added a commit that referenced this pull request Aug 12, 2026
#242 landed `Scenario: 14` in model_serving.feature while this branch was also
using 14, and the two touch different parts of the file — so git merges them
cleanly and the collision is invisible until someone tries to refer to
"scenario 14" and finds two of them.

Renumbers this branch's five additions to 15-19, after main's highest. No
scenario, tag, step or expectation changes.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fredespi added a commit that referenced this pull request Aug 13, 2026
#242 landed `Scenario: 14` in model_serving.feature while this branch was also
using 14, and the two touch different parts of the file — so git merges them
cleanly and the collision is invisible until someone tries to refer to
"scenario 14" and finds two of them.

Renumbers this branch's five additions to 15-19, after main's highest. No
scenario, tag, step or expectation changes.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fredespi added a commit that referenced this pull request Aug 13, 2026
#242 landed `Scenario: 14` in model_serving.feature while this branch was also
using 14, and the two touch different parts of the file — so git merges them
cleanly and the collision is invisible until someone tries to refer to
"scenario 14" and finds two of them.

Renumbers this branch's five additions to 15-19, after main's highest. No
scenario, tag, step or expectation changes.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mikeroySoft pushed a commit to mikeroySoft/rocm-cli that referenced this pull request Aug 27, 2026
`serve-hf-checkpoint-inference` has failed on every Strix Halo Windows
run since it was introduced in ROCm#242: the managed lemonade serve of
`unsloth/Qwen3-0.6B-GGUF:Q4_0` through the `owner/repo:variant` direct
path exits 0, but `/v1/models` never lists the model. It is the lane's
only host-independent unexpected failure, so the lane reads red on every
open PR and a genuine new Windows regression is easy to miss.

Declare it as the known bug it is, scoped to Windows GPU hosts only: the
Strix Halo Ubuntu lane serves the same checkpoint fine, and on Windows
the short-recipe-name path still passes, so a wider row would mask real
regressions on both.

Also shorten the serve wait to 240s. The lane leaves
E2E_SERVE_TIMEOUT_SECS unset, so today it burns the full 600s default on
a serve that never arrives. A healthy serve of this cache-warm
checkpoint reaches ready in ~120s on this host, so 240s keeps ~2x
headroom -- unlike the 90s the EAI-7423 rows use, which would sit below
a healthy serve and hide the fix by keeping the scenario red after the
bug is closed.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
fredespi added a commit that referenced this pull request Sep 15, 2026
#242 landed `Scenario: 14` in model_serving.feature while this branch was also
using 14, and the two touch different parts of the file — so git merges them
cleanly and the collision is invisible until someone tries to refer to
"scenario 14" and finds two of them.

Renumbers this branch's five additions to 15-19, after main's highest. No
scenario, tag, step or expectation changes.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants