Skip to content

rocm serve --engine lemonade fails on any headless host where XDG_RUNTIME_DIR is unset #259

Closed

Description

@rominf

Symptom

On a Linux host with no XDG_RUNTIME_DIR — a systemd service, a cron job, a container,
an SSH session without lingering enabled — starting the lemonade engine fails:

$ rocm serve Qwen3-0.6B-GGUF --engine lemonade --managed
Preparing lemonade for GPU serving...
Preparing Lemonade embeddable 11.5.1...
Detecting best supported Lemonade llama.cpp backend...
Error: request_failed: Lemonade server did not become ready: exit status: 1;
Lemonade startup log tail (.../engines/lemonade/runtime/install-lemond.log):
[Error] (main) Error: Unable to resolve writable runtime directory from
XDG_RUNTIME_DIR or RUNTIME_DIRECTORY

Exit code 1, every time. It is fully deterministic — retrying does not help — and it does
not depend on the model, the variant, or the serve path: both the recipe-name and
owner/repo:variant paths fail identically.

The same command works from an interactive desktop or SSH login on the same machine,
because a login session gets XDG_RUNTIME_DIR from pam_systemd. That makes this look
like a machine-specific quirk when it is really a property of how the process was started.

Root cause

lemond requires a writable runtime directory and looks for it in XDG_RUNTIME_DIR or
RUNTIME_DIRECTORY. Neither is set in a headless context: XDG_RUNTIME_DIR is populated
by pam_systemd at login and is absent for non-login processes, and RUNTIME_DIRECTORY
only exists for a systemd unit that declares RuntimeDirectory=.

rocm-cli launches lemond and builds its environment explicitly, but does not supply
either variable. lemonade_process_environment_vars
(engines/lemonade/src/lib.rs:2170) already translates ROCM_PATH, PATH,
LD_LIBRARY_PATH, HIP_VISIBLE_DEVICES and LEMONADE_API_KEY into the child — the
runtime directory is simply missing from that list. XDG_RUNTIME_DIR appears nowhere in
the engine code; the only occurrences in the repo are the dashboard socket resolvers and
the E2E harness.

So the hard requirement belongs to lemond, but the thing that can satisfy it is
rocm-cli, which is what spawns the process and already owns its environment.

This is a product bug, not a CI bug

It was found on a self-hosted MI300X runner (see #247 for the full trail — it was masked
until now by the Lemonade embeddable's glibc floor, tracked separately in #258),
but the runner is just an ordinary headless host. Anyone in these situations hits it:

  • running rocm serve from a systemd unit without RuntimeDirectory=
  • running it from cron, or any other non-login context
  • running it inside a container that has no systemd session
  • ssh host 'rocm serve ...' where the user has no lingering session
  • any CI, self-hosted or otherwise

Setting XDG_RUNTIME_DIR in the workflow would turn the lane green and leave every one of
those users broken, so please don't fix it there.

The E2E harness detail is worth stating for the same reason: pty_env
(tests/e2e-cucumber/tests/e2e.rs:240) sets HOME and XDG_RUNTIME_DIR, but only for PTY
scenarios — its comment says "Piped scenarios keep their historical HOME/XDG environment",
and the serve scenarios are piped, so they inherit the host's environment. That is
correct behaviour and should stay: it is precisely why the suite reproduces the real
headless case instead of hiding it.

Suggested fix — reuse the precedence this codebase already has

The project has already solved "pick a writable, user-owned runtime directory, with a
sensible fallback chain" once, for the dashboard socket. dashboard_socket_path
(crates/rocm-core/src/lib.rs:5269) is a pure function taking its env inputs as
arguments, with a documented three-tier precedence:

  1. $XDG_RUNTIME_DIR — already mode 0700 on systemd systems, ideal.
  2. $HOME/.rocm/data/telemetry — standard per-user data dir.
  3. temp_dir()/rocm-<user> — user-named subdir, so the parent is something we create and
    own rather than /tmp itself, with the user name sanitized so a path separator or
    .. cannot escape it.

It has three unit tests covering each tier, empty-value fallthrough and sanitization
(crates/rocm-core/src/lib.rs:7483 onward), and the precedence is deliberately mirrored
in rocm-dash-core with matching tests.

The proposal is to apply the same precedence — not a new mechanism — when launching the
engine: if XDG_RUNTIME_DIR is absent from the parent environment, resolve a directory by
that chain, create it 0700, and pass it to the child in
lemonade_process_environment_vars. That function is already the single place where child
environment is assembled, so the change is local, and following the existing pattern keeps
one answer to this question in the codebase rather than two.

Worth deciding as part of the fix:

  • Whether the resolved directory should be shared with the dashboard's or kept distinct
    (a rocm-<user>/lemonade subdir) so an engine crash cannot disturb telemetry.
  • Whether to factor the tier chain into a small shared helper now that it would have a
    third caller, rather than a third copy.
  • Whether to set XDG_RUNTIME_DIR for the child or only pass a directory by another
    route — the former is what lemond reads, but it also affects anything else the child
    spawns.

A unit test on lemonade_process_environment_vars asserting the variable is present when
absent from the parent environment, and passed through unchanged when present, would cover
this at the level where it broke. lemonade_process_environment_vars has no tests today.

Reproduction

env -u XDG_RUNTIME_DIR -u RUNTIME_DIRECTORY rocm serve Qwen3-0.6B-GGUF --engine lemonade --managed

In CI, on the MI300X GPU lane after its 24.04 reimage:
https://github.com/ROCm/rocm-cli/actions/runs/31679587566/job/94711980880 — the error
appears on both lemonade serves in the job.

Note the stderr is only visible because that job runs #248's diagnostics change; on main
today the same failure prints an empty message.

Environment

  • Affects Linux hosts (native and WSL2) where the CLI runs outside a login session
  • lemonade pinned at 11.5.1 (runtime-deps.toml)
  • Not observed on the Strix Halo Ubuntu lane, whose lemonade serves pass — consistent
    with that host having a login session, though I have not confirmed its environment
    directly
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions