Repository navigation
ROCMAI-898: resolve registered Lemonade model names when serving directly on Linux - #963
Conversation
…ctly on Linux On Linux, once a packaged llama-server backend is installed, `rocm serve <model> --engine lemonade` loads the model through a short-lived lemond, then looks for its GGUF in the Hugging Face cache to launch llama-server directly. That lookup only knew the built-in default model and `owner/repo:variant` refs, so for any other name Lemonade registers (`Qwen3-0.6B-GGUF`, or an alias such as `qwen-smoke`) it killed the loaded server and bailed with "did not download", even with the GGUF on disk. The command exited 0 with `readiness: starting` and the endpoint never served the model. Resolve a registered name to its checkpoint through the `resources/server_models.json` registry shipped in the Lemonade embeddable, validated like any other checkpoint, and look that up. A checkpoint matching more than one cached file is not guessed at. The runtime directory is now a required argument of the lookup, so no caller can fall back to one that cannot see registered names, and the error text is derived from the same lookup instead of claiming nothing was downloaded. Verified on a Strix Halo (gfx1151) box in an isolated root: on main the e2e scenarios serve-07, serve-40, chat-07 and bench-06 fail; with this change they pass, and a chat request returns a reply. Signed-off-by: juhovainio <juho.vainio@amd.com>
jussielo-amd
left a comment
There was a problem hiding this comment.
Looks good. This was a static review of the diff; I did not run the tests. CI on the head commit is green apart from the W7900 lane, which was still queued when I looked.
The fix does what the description says: registered names and aliases go through the server_models.json registry, the checkpoint is validated by parse_hf_checkpoint (so a registry entry cannot climb out of the cache), the runtime dir is now a required argument at all call sites, and several matches are not guessed at. No blockers.
Non-blocking, beyond the inline comments:
- The description calls
Qwen3-0.6B-GGUFthe built-in default, butDEFAULT_MODELinlib.rsisQwen3-4B-Instruct-2507-GGUF;Qwen3-0.6B-GGUFis a registered model. The doc comment ona_registry_checkpoint_naming_an_exact_file_resolves_to_that_filerepeats this ("how the built-in default model is registered"), but the default resolves throughfind_default_qwen_ggufand never reaches the registry. docs/architecture.mdstill describesdirect_llama.rsas checkpoint/GGUF resolution and the direct serve path; it now also reads Lemonade'sserver_models.json, so the entry prose could mention that.- The PR title does not follow the Conventional Commits format CONTRIBUTING.md asks for (e.g.
fix(lemonade): ...).
🤖 by agenthub on AMD AgentHub
| // expected case. Several (a `Q4_0` label also naming an `IQ4_0` file, say) cannot | ||
| // be told apart from here, and serving a different quantization than the one | ||
| // registered is worse than not finding the file. | ||
| if matches.len() == 1 { |
There was a problem hiding this comment.
Non-blocking: requiring exactly one match makes some registered models unservable. filter_ggufs_by_variant matches by substring, so a BF16/F16 label also matches an mmproj-BF16.gguf, and a Q4_K_M label matches every shard of a split model (-00001-of-0000N.gguf). Both end up here as None, and the resulting message tells the user to serve owner/repo:<file>.gguf, which does not work for shards. You list shards as not verified, so this may be fine for now; consider skipping mmproj-* files in this path and treating the shards of one set as a single match (first shard), or at least a test plus a message that does not suggest a per-file ref for shards.
| } | ||
| } | ||
| format!( | ||
| "no cached GGUF was found for `{model_ref}`, so direct ROCm serving cannot start (searched {})", |
There was a problem hiding this comment.
Non-blocking: this fallback also covers "the registry is missing/unreadable" and "the name is not in the registry" (registry_checkpoint returns None silently for both), and ensure_direct_llama_model_available discards the run_lemonade_model_load result with let _ =. A mistyped model name therefore gets a cache-directory listing and no hint that it is not a registered model, which is the very thing this PR is making the message honest about. Consider a "not registered with Lemonade" branch, or appending the load error. The sibling message in serve_direct_llama_server ("was not found for direct serving") is built separately and can still be wrong in the several-files case; building it from unresolved_model_message would fix the whole class (AGENTS.md §3).
| /// must pass the runtime's registry on to the lookup: found here only through the | ||
| /// registry, in the cache directory the CLI itself owns. | ||
| #[test] | ||
| fn the_entry_point_threads_the_runtime_registry_through() { |
There was a problem hiding this comment.
Non-blocking: this test goes through direct_llama_model_path, which uses the real hf_cache_roots (env vars and $HOME) alongside the temp cache_dir. On a developer machine that already has Qwen3-0.6B-Q4_0.gguf (or an IQ4_0 sibling) cached under HF_HOME/~/.cache, the returned path or the single-match rule can differ and the assertion would fail. The other tests in this module pass explicit roots; driving hf_cache_roots_from with injected env/home closures would keep this one hermetic too.
#963 fixed Lemonade serving on Linux by resolving registered model names, but the ROCMAI-898 expected-failure rows stayed in expectations.toml. Every one of them is flaky, so their passes were tolerated as XPASS and never surfaced as stale rows. The lanes have been passing these scenarios since. In the main push and merge-queue E2E runs from #963's merge (2026-11-09) to 2026-11-13, the Strix Halo Ubuntu jobs passed every removed scenario that ran, MI300X passed serve-07 and serve-40 (as it also did on this PR's own MI300X job), and the W7900 jobs in that window passed them too. The dash-lemonade-managed-inference-journey row, added by #631, has passed in all seven Strix Halo Ubuntu jobs since that change merged. Remove twelve rows: the default-engine, readiness-contract, Anthropic inference/tool-use/streaming, explicit Lemonade inference (serve-07) and background-helper (serve-40) serving scenarios, the two chat scenarios, the bench real-serve scenario and the dash managed-inference journey. Keep serve-50 and serve-52 as flaky rows, re-attributed to ROCMAI-1023: their serve becomes ready, but `rocm services list` intermittently shows no `anthropic endpoint:` row within the 30s poll. They set no serve-timeout override, since the serve itself succeeds. Add a guard test that resolves each of the twelve scenarios from its real tag line on the modelled Linux Lemonade hosts (Strix Halo Ubuntu, a WSL2 guest with a usable GPU, gfx1100 bare metal) and, for the ids pinned to Lemonade, on MI300X, failing if any row matches; the pin is held per id and the failure message says how to proceed. Tests that used the deleted rows as fixtures now parse inline matrices. Update docs/testing.md and a model_serving.feature comment that still described the old rows. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Why
On Linux,
rocm serve <model> --engine lemonade --managedexits 0 withreadiness: startingand the endpoint never serves the model, for every Lemonade-registered model other than the built-in default (Qwen3-0.6B-GGUF, or an alias such asqwen-smoke). The service log ends withLemonade did not download <model> for direct ROCm serving, even when the GGUF is already in a searched Hugging Face cache root.The cause: once a packaged llama-server backend is installed, the Linux path loads the model through a short-lived lemond, then looks for its GGUF in the Hugging Face cache to launch llama-server directly.
direct_llama_model_pathonly knew the default model andowner/repo:variantrefs, so for any other registered name it killed the loaded server and bailed. This is what the nightly E2E lanes show on MI300X and W7900 (serve-07, serve-40, chat-07, bench-06), previously tracked as flaky under EAI-7423 and #897.What changes
A registered name (or alias) is resolved to its checkpoint through the
resources/server_models.jsonregistry in the Lemonade embeddable, validated by the sameparse_hf_checkpointas any checkpoint, and that checkpoint's GGUF is looked up in the cache. The runtime directory is now a required argument of the lookup, so no caller can fall back to one that cannot see registered names. A checkpoint matching more than one cached file is not guessed at: serving a different quantization than the registered one is worse than failing. The error text is derived from the same lookup, so it says which of "several files match" or "none found" happened instead of claiming nothing was downloaded.Verification
Reproduced on a Strix Halo (gfx1151, WSL2) box in an isolated root with Lemonade 11.5.1 and SDK 7.14.1, and verified with the real harness (
cargo xtask e2e -- -n ...):mainserve-lemonade-inference)serve-stop-leaves-no-background-helper)chat-end-to-end-local-model)bench-load-real-serve)A
rocm chat --provider localrequest against the fixed serve returns a reply. The scenarios above are@requires-gpu @requires-engine:lemonadeand run on the Linux GPU lanes; they already cover this behaviour, so no scenario is added.Unit tests cover resolution by name and alias, an uncached model, an unregistered name, malformed registries, a registry checkpoint that tries to climb out of the cache, the single-match rule, the entry point threading the registry through, and the message following the reason the lookup failed. Each guard was broken in turn (single-match rule, checkpoint validation, alias resolution, registry threading, message branch) and its test went red. The fixture entries are trimmed from the real 11.5.1
server_models.json.The per-PR hardware lanes show the same result. On Strix Halo Ubuntu (release and nightly) serve-07, serve-40, chat-06, chat-07 and bench-06 all pass, and on MI300X serve-07, serve-40, chat-07 and bench-06 pass; the existing
flakyxfail rows tolerate these passes, and the same Strix Ubuntu lane on a branch without this change fails them with the symptom above.Not verified: the W7900 hardware lane, and a sharded GGUF, which a registry variant can match as several files and which therefore resolves to nothing rather than the first shard.
Scope
The xfail rows for these scenarios (EAI-7423, #897) are
flaky = true, so they tolerate the new passes; they are left in place and should be deleted once the lanes confirm. Nothing here touches Windows, where registered names go through the router path.tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. (None carry this ticket ID; the relatedflakyrows are intentionally kept, see Scope.)docs/architecture.md. (Not applicable.)docs/architecture.mdhas the matching entry. (No files added.)