Skip to content

local_assistant_therock_gpu_test.py: direct llama-server mode never detected when service manifest is already ready → false timeout #389

Open

Description

@mikeroySoft

Current behavior

scripts/local_assistant_therock_gpu_test.py fails every live run against a Lemonade service that uses the direct lemonade-packaged-llama-server backend, even though the service is ready and answering inference. It waits the full --timeout polling /v1/models, then raises:

RuntimeError: managed service endpoint did not report the requested loaded model before timeout: model was not loaded yet; HTTP 200 from /v1/models; endpoint=http://127.0.0.1:11437/v1

Meanwhile the product side is healthy: rocm serve prints readiness: ready, the engine state file has backend_requested: rocm-stable, backend_mode: lemonade-packaged-llama-server, load_response.status: loaded, and the CLI's own inference probe succeeded (inference_verified_at_unix_ms set). A manual POST /v1/chat/completions against the same endpoint returns a correct answer on the GPU.

Expected behavior

The harness should recognize the direct llama-server mode (which the script already has a code path for, lemonade_direct_rocm) and accept the /v1/models entry that names the model, instead of requiring recipe_options.llamacpp_backend, which a bare llama-server never emits.

Steps to reproduce

On a Linux host with a ready managed TheRock runtime and the release binary built:

python scripts/local_assistant_therock_gpu_test.py \
  --rocm target/release/rocm --engine lemonade --model qwen --port 11437 --timeout 90 \
  --runtime-id <active runtime key>

Observed on a6fa76dbca5d767c1190ee47b765d16bad0e4a53 (current main), twice (20-minute and 90-second timeouts). Because the packaged llama-server becomes ready in ~2 s, the service manifest is already status: ready by the time the script reads it, so the bug triggers deterministically.

Root cause (script)

wait_ready_manifest_path (line 395) returns as soon as the service manifest says status: ready, before the branch that loads the engine state file and attaches it as manifest["engine_state"] (lines 397-406). wait_local_endpoint (lines 463-468) then computes

lemonade_direct_rocm = engine == "lemonade" and str(
    engine_state.get("backend_requested") or manifest.get("backend_requested") or ""
).strip().lower().startswith("rocm")

with engine_state == {} and no backend_requested on the service manifest → False → require_rocm_backend=True for the /v1/models probe → payload_entry_reports_rocm_backend never matches the plain llama-server payload:

{"object":"list","data":[{"id":"Qwen3-4B-Instruct-2507-GGUF","aliases":["Qwen3-4B-Instruct-2507-GGUF"],"object":"model","owned_by":"llamacpp","meta":{...}}]}

The engine state file that would have flipped the flag:

{"backend_requested":"rocm-stable","backend_mode":"lemonade-packaged-llama-server","status":"ready","load_response":{"status":"loaded","llamacpp_backend":"rocm-stable"}}

Possible solution

In wait_ready_manifest_path, always load engine_state_path when it exists and attach it to the returned manifest, regardless of whether the service manifest is already ready (i.e. move the engine-state read above the early return, or fall through to it). Alternatively have wait_local_endpoint read engine_state_path itself when manifest["engine_state"] is absent.

Your environment

  • rocm-cli channel: nightly (TheRock 10.1.0a20260822, gfx120X-all)
  • rocm-cli version (rocm --version): rocm 0.1.0, built from a6fa76dbca5d767c1190ee47b765d16bad0e4a53
  • Platform: Linux
  • OS / Distro: CachyOS (Arch), kernel 7.2
  • GPU/APU: AMD Radeon AI PRO R9700 (gfx1201); second GPU AMD Radeon RX 5700 XT (gfx1010)
  • Engine env: lemonade-embeddable-11.5.1, backend rocm-stable, llama-server b9752

Relevant log output

serve plan
  requested model: qwen
  resolved model: Qwen3-4B-Instruct-2507-GGUF
  engine: lemonade
  runtime_id: adopted-nightly-pip-10-1-0a20260822-gfx120x-all-10-1-0a20260822
  selection_source: cli_runtime_id
  device_policy: gpu_required
  gpu: auto (selected 1)
managed service launched
  service_id: lemonade-qwen3-4b-instruct-2507-g-1789163526848
  endpoint: http://127.0.0.1:11437/v1
  readiness: ready
[local-assistant-gpu-test] Stopping managed service lemonade-qwen3-4b-instruct-2507-g-1789163526848.
Local server stopped
  stopped processes: 1
RuntimeError: managed service endpoint did not report the requested loaded model before timeout: model was not loaded yet; HTTP 200 from /v1/models; endpoint=http://127.0.0.1:11437/v1

Service log (product side, same run):

Launching Lemonade packaged rocm-stable llama-server directly: using Lemonade packaged rocm-stable llama-server directly on Linux
0.00.102.614 I   - ROCm0   : AMD Radeon RX 5700 XT (8176 MiB, 8058 MiB free)
0.01.990.717 I srv  llama_server: model loaded
0.01.990.719 I srv  llama_server: server is listening on http://127.0.0.1:11437

Additional context

Side observation from the same run, not the cause of this failure: --gpu auto selected GPU 1 (RX 5700 XT, gfx1010) because it had more free VRAM than the busy gfx1201 card, even though the active runtime family is gfx120X-all and only rocm-sdk-device-gfx1201 is installed. Lemonade's packaged llama-server carries its own gfx1010 code objects so it worked, but the harness has no --gpu passthrough to pin the validation GPU. Related ComfyUI harness issue: #390. Running the documented direct path (rocm serve … --gpu 0) and issuing a chat request manually passed on gfx1201.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions