Repository navigation
Description
Symptom
On a Linux host with no XDG_RUNTIME_DIR — a systemd service, a cron job, a container,
an SSH session without lingering enabled — starting the lemonade engine fails:
$ rocm serve Qwen3-0.6B-GGUF --engine lemonade --managed
Preparing lemonade for GPU serving...
Preparing Lemonade embeddable 11.5.1...
Detecting best supported Lemonade llama.cpp backend...
Error: request_failed: Lemonade server did not become ready: exit status: 1;
Lemonade startup log tail (.../engines/lemonade/runtime/install-lemond.log):
[Error] (main) Error: Unable to resolve writable runtime directory from
XDG_RUNTIME_DIR or RUNTIME_DIRECTORY
Exit code 1, every time. It is fully deterministic — retrying does not help — and it does
not depend on the model, the variant, or the serve path: both the recipe-name and
owner/repo:variant paths fail identically.
The same command works from an interactive desktop or SSH login on the same machine,
because a login session gets XDG_RUNTIME_DIR from pam_systemd. That makes this look
like a machine-specific quirk when it is really a property of how the process was started.
Root cause
lemond requires a writable runtime directory and looks for it in XDG_RUNTIME_DIR or
RUNTIME_DIRECTORY. Neither is set in a headless context: XDG_RUNTIME_DIR is populated
by pam_systemd at login and is absent for non-login processes, and RUNTIME_DIRECTORY
only exists for a systemd unit that declares RuntimeDirectory=.
rocm-cli launches lemond and builds its environment explicitly, but does not supply
either variable. lemonade_process_environment_vars
(engines/lemonade/src/lib.rs:2170) already translates ROCM_PATH, PATH,
LD_LIBRARY_PATH, HIP_VISIBLE_DEVICES and LEMONADE_API_KEY into the child — the
runtime directory is simply missing from that list. XDG_RUNTIME_DIR appears nowhere in
the engine code; the only occurrences in the repo are the dashboard socket resolvers and
the E2E harness.
So the hard requirement belongs to lemond, but the thing that can satisfy it is
rocm-cli, which is what spawns the process and already owns its environment.
This is a product bug, not a CI bug
It was found on a self-hosted MI300X runner (see #247 for the full trail — it was masked
until now by the Lemonade embeddable's glibc floor, tracked separately in #258),
but the runner is just an ordinary headless host. Anyone in these situations hits it:
- running
rocm servefrom a systemd unit withoutRuntimeDirectory= - running it from cron, or any other non-login context
- running it inside a container that has no systemd session
ssh host 'rocm serve ...'where the user has no lingering session- any CI, self-hosted or otherwise
Setting XDG_RUNTIME_DIR in the workflow would turn the lane green and leave every one of
those users broken, so please don't fix it there.
The E2E harness detail is worth stating for the same reason: pty_env
(tests/e2e-cucumber/tests/e2e.rs:240) sets HOME and XDG_RUNTIME_DIR, but only for PTY
scenarios — its comment says "Piped scenarios keep their historical HOME/XDG environment",
and the serve scenarios are piped, so they inherit the host's environment. That is
correct behaviour and should stay: it is precisely why the suite reproduces the real
headless case instead of hiding it.
Suggested fix — reuse the precedence this codebase already has
The project has already solved "pick a writable, user-owned runtime directory, with a
sensible fallback chain" once, for the dashboard socket. dashboard_socket_path
(crates/rocm-core/src/lib.rs:5269) is a pure function taking its env inputs as
arguments, with a documented three-tier precedence:
$XDG_RUNTIME_DIR— already mode0700on systemd systems, ideal.$HOME/.rocm/data/telemetry— standard per-user data dir.temp_dir()/rocm-<user>— user-named subdir, so the parent is something we create and
own rather than/tmpitself, with the user name sanitized so a path separator or
..cannot escape it.
It has three unit tests covering each tier, empty-value fallthrough and sanitization
(crates/rocm-core/src/lib.rs:7483 onward), and the precedence is deliberately mirrored
in rocm-dash-core with matching tests.
The proposal is to apply the same precedence — not a new mechanism — when launching the
engine: if XDG_RUNTIME_DIR is absent from the parent environment, resolve a directory by
that chain, create it 0700, and pass it to the child in
lemonade_process_environment_vars. That function is already the single place where child
environment is assembled, so the change is local, and following the existing pattern keeps
one answer to this question in the codebase rather than two.
Worth deciding as part of the fix:
- Whether the resolved directory should be shared with the dashboard's or kept distinct
(arocm-<user>/lemonadesubdir) so an engine crash cannot disturb telemetry. - Whether to factor the tier chain into a small shared helper now that it would have a
third caller, rather than a third copy. - Whether to set
XDG_RUNTIME_DIRfor the child or only pass a directory by another
route — the former is whatlemondreads, but it also affects anything else the child
spawns.
A unit test on lemonade_process_environment_vars asserting the variable is present when
absent from the parent environment, and passed through unchanged when present, would cover
this at the level where it broke. lemonade_process_environment_vars has no tests today.
Reproduction
env -u XDG_RUNTIME_DIR -u RUNTIME_DIRECTORY rocm serve Qwen3-0.6B-GGUF --engine lemonade --managed
In CI, on the MI300X GPU lane after its 24.04 reimage:
https://github.com/ROCm/rocm-cli/actions/runs/31679587566/job/94711980880 — the error
appears on both lemonade serves in the job.
Note the stderr is only visible because that job runs #248's diagnostics change; on main
today the same failure prints an empty message.
Environment
- Affects Linux hosts (native and WSL2) where the CLI runs outside a login session
- lemonade pinned at
11.5.1(runtime-deps.toml) - Not observed on the Strix Halo Ubuntu lane, whose lemonade serves pass — consistent
with that host having a login session, though I have not confirmed its environment
directly
Activity