Skip to content

fix(lemonade): give the server a runtime dir when XDG_RUNTIME_DIR is unset - #262

Merged
michaelroy-amd merged 5 commits into
mainfrom
fix-lemonade-missing-xdg-runtime-dir
Aug 18, 2026
Merged

michaelroy-amd merged 5 commits into
mainfrom
fix-lemonade-missing-xdg-runtime-dir

Conversation

@rominf

@rominf rominf commented Aug 14, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #259.

Symptom

rocm serve --engine lemonade --managed fails on any Linux host with no
XDG_RUNTIME_DIR. It is fully deterministic and independent of the model or the serve
path:

Error: request_failed: Lemonade server did not become ready: exit status: 1;
Lemonade startup log tail (.../engines/lemonade/runtime/install-lemond.log):
[Error] (main) Error: Unable to resolve writable runtime directory from
XDG_RUNTIME_DIR or RUNTIME_DIRECTORY

The same command works from an interactive desktop or SSH login on the same machine, so
it reads as a machine quirk when it is really a property of how the process was started:
a systemd unit, a cron job, a container, ssh host 'rocm serve ...' without lingering,
or any CI runner.

Root cause

The message comes from the spawned server, not from rocm-cli — the string appears nowhere
in this tree and RUNTIME_DIRECTORY is never referenced here. The server needs a writable
runtime directory and consults exactly two sources, neither of which exists in a headless
context:

  • XDG_RUNTIME_DIR, populated by pam_systemd at login and therefore absent for every
    non-login process;
  • RUNTIME_DIRECTORY, which only exists for a systemd unit declaring RuntimeDirectory=.

lemonade_process_environment_vars is the single place the child environment is
assembled. It translated ROCM_PATH, PATH, LD_LIBRARY_PATH, HIP_VISIBLE_DEVICES and
LEMONADE_API_KEY, but nothing about the runtime directory. rocm-cli cannot change what
the server requires, but it owns the environment it hands over, so this is the layer that
can satisfy it.

Fix

Reuse the precedence this repo already has rather than inventing a second one.
dashboard_socket_path already solves "pick a writable, user-owned runtime directory with
a fallback chain", as a pure function taking its env inputs as arguments:

  1. $XDG_RUNTIME_DIR
  2. $HOME/.rocm/data/…
  3. temp_dir()/rocm-<user>, with the user name sanitized so a separator or .. cannot
    escape the subdirectory

That tier chain moves into rocm_core::user_runtime_dir; dashboard_socket_path becomes
a thin caller over it and resolves to byte-identical paths in all three tiers (its three
existing tests are unchanged and still pass). The engine calls the same helper when the
parent environment has no usable value, creates the directory mode 0700 before spawning,
and passes it down.

Why not fix this in CI

A workflow-level fix was considered and rejected. Exporting XDG_RUNTIME_DIR in
.github/workflows/e2e-selfhosted.yml, or setting it for piped scenarios in the E2E
harness, would turn the GPU lane green and leave every headless user exactly as broken.

pty_env (tests/e2e-cucumber/tests/e2e.rs) does set HOME and XDG_RUNTIME_DIR, but
only for PTY scenarios — its comment records that piped scenarios keep their historical
HOME/XDG environment, and the serve scenarios are piped. That is correct and stays: it is
precisely why the suite reproduces the real headless case instead of hiding it.

Decisions worth reviewing

A distinct directory for the engine, not the dashboard's. In the fallback tiers the
engine gets a lemonade leaf ($HOME/.rocm/data/lemonade,
temp_dir()/rocm-<user>/lemonade). Sharing one directory would put engine scratch state
next to the telemetry socket, so a crashing or misbehaving server could disturb the
dashboard. The isolation costs one path component.

A shared helper, but only for the two callers that can share one. The tier chain is
extracted into rocm-core, used by dashboard_socket_path and by the engine, so there
are still two copies of this logic rather than three. rocm-dash-core keeps its own
mirror deliberately: it is a standalone library with a lean dependency set, and depending
on rocm-core to share ~30 lines would pull in that crate's whole graph. Its "keep the
two in sync" comment now names the shared helper so the sync target is unambiguous, and
the mirrored tests on both sides still catch divergence.

XDG_RUNTIME_DIR rather than RUNTIME_DIRECTORY. The server reads either.
RUNTIME_DIRECTORY is narrower, but it is a systemd-unit contract implying systemd
created the directory and will clean it up — untrue here, and misleading to anyone
debugging. XDG_RUNTIME_DIR is the freedesktop-standard variable whose absence is the bug;
supplying a valid one is strictly better for any descendant process than leaving it unset.
It is never overridden: a value already in the parent environment is passed through
unchanged.

Hardening. The tier-3 fallback lands under a shared temp dir, so a pre-existing path
there may not be ours. A symlink is refused rather than chmod-ed through, and a chmod
failure is a hard error naming the path.

Verification

Ran locally on a headless Linux host (no XDG_RUNTIME_DIR, no RUNTIME_DIRECTORY):

  • cargo fmt --all --check
  • cargo clippy --locked --workspace --all-targets -- -D warnings
  • cargo clippy --locked -p e2e-cucumber --test e2e -- -D warnings
  • cargo test --workspace --all-targets, plus the existing dashboard_socket_path and
    rocm-dash-core mirror tests
  • driving the real resolver against this host's actual environment: it produced
    $HOME/.rocm/data/lemonade, created drwx------

Five new unit tests on lemonade_process_environment_vars cover synthesis when the parent
has no value, an exported-but-empty value, the no-HOME tier, pass-through of an existing
value, and refusal of a symlinked directory. All five fail before this change and pass
after. They take the environment as arguments rather than mutating process-global env, so
they behave identically under cargo test and nextest.

Re-verified after rebasing onto current main: all five gates above pass, and
cargo test --workspace --all-targets is now fully green. Two shared-process cargo test
failures noted here earlier
(therock::tests::extracting_the_sdk_archive_removes_it,
providers::tests::local_provider_default_chat_requires_builtin_qwen_assistant) were
pre-existing on main and are gone since #253 landed; nothing about them was ever related
to this change.

The real verification is the GPU lane, and it is now green. No GPU here, so a full
managed serve could not be run locally; @id:serve-lemonade-inference and
@id:serve-hf-checkpoint-inference (@requires-gpu @requires-engine:lemonade) exercise
this on E2E tests (GPU), the MI300X lane where the failure reproduces.

before (job, the run in #259) after (job)
"Unable to resolve writable runtime directory" in the log 2 occurrences none
serve-hf-checkpoint-inference failed passes
reconciliation 1 unexpected failure — lane red 0 unexpected failures — lane green

serve-lemonade-inference still xfails, but on the unrelated EAI-7423 symptom it is
already registered for ("did not serve model within 90s"), and it now gets past server
startup instead of exiting immediately.

Worth being explicit about the shape of that coverage: the scenarios are piped, so they
inherit the runner's environment rather than asserting on it — they regress this bug
because that runner is headless, not because the harness forces it. Making it explicit
would mean touching the piped scenarios' environment, which is the thing this fix
deliberately avoids.

tests/e2e-cucumber/expectations.toml was searched: no row describes this failure mode.
The lemonade Linux rows there are EAI-7423 ("reaches ready then shuts down immediately") on
the Strix Halo Ubuntu lane, a different symptom on a different host, and are left alone.

…unset

`rocm serve --engine lemonade --managed` failed deterministically on any
Linux host without XDG_RUNTIME_DIR: the spawned server exits during
startup with "Unable to resolve writable runtime directory from
XDG_RUNTIME_DIR or RUNTIME_DIRECTORY". XDG_RUNTIME_DIR is populated by
pam_systemd at login, so it is absent for every non-login process (cron
jobs, CI runners, systemd-run, a bare container exec), and
RUNTIME_DIRECTORY only exists for a systemd unit declaring
RuntimeDirectory=.

rocm-cli assembles the child environment, so it owns the seam. Reuse the
runtime-directory precedence the dashboard socket already uses
(XDG_RUNTIME_DIR, then $HOME/.rocm/data, then temp_dir()/rocm-<user>) by
extracting its tier chain into rocm-core as `user_runtime_dir`, and give
the engine a `lemonade` leaf inside the fallback tiers so a crashing
server cannot disturb the telemetry socket sharing the same root. The
directory is created mode 0700 before the server is spawned, and a
symlink there is refused rather than chmod'ed through. A value already
present in the parent environment is passed through unchanged.

The readiness poll re-spawns a status probe twice a second, so it now
resolves the child environment once instead of repeating the directory
preparation on every attempt.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf marked this pull request as ready for review August 14, 2026 13:25
@rominf
rominf requested a review from a team as a code owner August 14, 2026 13:25
@rominf
rominf marked this pull request as draft August 14, 2026 13:46
Signed-off-by: Michael Roy <michael.roy@amd.com>
…unset

`rocm serve --engine lemonade --managed` failed deterministically on any
Linux host without XDG_RUNTIME_DIR: the spawned server exits during
startup with "Unable to resolve writable runtime directory from
XDG_RUNTIME_DIR or RUNTIME_DIRECTORY". XDG_RUNTIME_DIR is populated by
pam_systemd at login, so it is absent for every non-login process (cron
jobs, CI runners, systemd-run, a bare container exec), and
RUNTIME_DIRECTORY only exists for a systemd unit declaring
RuntimeDirectory=.

rocm-cli assembles the child environment, so it owns the seam. Reuse the
runtime-directory precedence the dashboard socket already uses
(XDG_RUNTIME_DIR, then $HOME/.rocm/data, then temp_dir()/rocm-<user>) by
extracting its tier chain into rocm-core as `user_runtime_dir`, and give
the engine a `lemonade` leaf inside the fallback tiers so a crashing
server cannot disturb the telemetry socket sharing the same root. The
directory is created mode 0700 before the server is spawned, and a
symlink there is refused rather than chmod'ed through. A value already
present in the parent environment is passed through unchanged.

The readiness poll re-spawns a status probe twice a second, so it now
resolves the child environment once instead of repeating the directory
preparation on every attempt.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the fix-lemonade-missing-xdg-runtime-dir branch from 7aa33bf to afd657b Compare August 17, 2026 12:22
@rominf
rominf marked this pull request as ready for review August 17, 2026 12:23
@rominf

rominf commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto current main and out of draft. Post-rebase CI, on the lane this PR is about:

E2E tests (GPU) is green — job.
That lane fails on main today with Unable to resolve writable runtime directory from XDG_RUNTIME_DIR or RUNTIME_DIRECTORY (twice per run), which is #259 and what this change fixes.

The one remaining red check, E2E tests (Strix Halo, Windows), is not this PR. It reports
exactly the same single unexpected failure as main does —
serve-hf-checkpoint-inference — which is #260, a Windows-only bug with a different cause and
symptom (exit 0 and never serves, versus exit 1 here):

39 scenarios (35 passed, 4 failed)
Reconciliation: 3 xfail, 0 XPASS, 1 unexpected failure(s).
FAIL: 'serve-hf-checkpoint-inference' was expected to pass on this host but FAILED

Identical to the baseline on main, so this PR neither causes nor fixes it. #260 is tracked
separately.

Everything else — all 16 required checks, plus the Strix Halo Ubuntu lane — is green.

@rominf

rominf commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto main (97ec0aa) and taken out of draft. All 16 required checks are green.

The one red check, E2E tests (Strix Halo, Windows), is not a required check and is not caused by this PR. Comparing against the main run for the very same base commit:

lane main @ 97ec0aa this PR @ afd657b
E2E tests (GPU) fails — 1 unexpected failure (serve-hf-checkpoint-inference), 2 occurrences of "Unable to resolve writable runtime directory" passes — 0 unexpected failures, 0 occurrences of that message
E2E tests (Strix Halo, Windows) fails fails

The Windows lane fails on both, and on a different symptom: serve-hf-checkpoint-inference reaching did not serve model Qwen3-0.6B within 600s, which is #260 and tracked separately. This change is Linux-only by construction — child_runtime_dir_var returns early unless runtime_is_linux() — so it cannot move that lane either way.

Re-verified locally after the rebase: cargo fmt --all --check, cargo clippy --locked --workspace --all-targets -- -D warnings, cargo clippy --locked -p e2e-cucumber --test e2e -- -D warnings, cargo test --workspace --all-targets (fully green now that #253 has landed), and python scripts/smoke_local.py.

Signed-off-by: Michael Roy <michael.roy@amd.com>
…f6cd

Signed-off-by: Michael Roy <michael.roy@amd.com>

@michaelroy-amd michaelroy-amd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Review focus: Please review this as filesystem security code: fallback precedence, descriptor-relative tier-3 creation, ownership/mode checks, and unchanged operator-provided XDG_RUNTIME_DIR behavior.
  • Problem / impact: On headless Linux processes without XDG_RUNTIME_DIR or RUNTIME_DIRECTORY, managed Lemonade exits before serving. CI runners, cron jobs, systemd-run invocations, and bare container execs therefore cannot use Lemonade. A naive /tmp fallback would add symlink, ownership, and path-replacement races.
  • State: #262, head 1f2ce58182f56b24e58fba6694733a5ef5b24bd4; OPEN, non-draft, MERGEABLE, REVIEW_REQUIRED (BLOCKED); 6 files, +1,172/-41.
  • Changed areas / review hotspots:
    1. crates/rocm-core/src/runtime.rs and lib.rs: shared precedence XDG_RUNTIME_DIR → $HOME/.rocm/data/<leaf> → <temp>/rocm-<sanitized-user>/<leaf>.
    2. crates/rocm-dash-core/src/config.rs: dashboard socket precedence remains behaviorally aligned without adding a heavy dependency.
    3. engines/lemonade/src/lib.rs: Linux-only child XDG_RUNTIME_DIR, lemonade isolation leaf, and one-time environment resolution reused by readiness probes.
    4. Tier-3 hardening: openat/mkdirat, O_NOFOLLOW, descriptor-relative creation, effective-UID ownership, mode 0700, and ancestry checks that prevent cross-UID entry replacement.
    5. Race/security regressions: symlinked leaves, foreign ownership, unsafe ancestors, temp-root replacement, user-parent replacement, and operator pass-through.
  • Invariants to verify: A non-empty parent XDG_RUNTIME_DIR passes through byte-for-byte with no fallback mutation; HOME and temp fallbacks end in a private lemonade leaf; user input cannot escape its path component; every created/opened component is no-follow, correctly owned, and private; creation stays anchored to the validated directory object if pathnames are swapped; failures are hard errors, never a permissive fallback; Windows behavior and strict GPU/no-CPU-fallback policy are unchanged.
  • Verification evidence: Required: all required hosted checks green, including Linux/Windows builds/tests, affected tests, clippy, signature+DCO, hygiene, and CodeQL. Advisory: GPU, Strix Ubuntu, and the self-hosted consolidated report pass—this is the only packet whose GPU leaf is green. Red: Strix Windows has five unexpected failures (bench-load-real-serve, chat-tool-definitions-accepted, chat-end-to-end-local-model, serve-lemonade-inference, serve-hf-checkpoint-inference) after the llama.cpp backend download exhausts six attempts with SSL connect error (CURL code: 35).
  • Recommended sequence: shared resolver → Lemonade child-env call sites → fd-relative hardening → adversarial tests → dashboard parity/platform gates.

@michaelroy-amd michaelroy-amd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at 1f2ce58182f56b24e58fba6694733a5ef5b24bd4. The focused review above covers the changed contracts and invariants. All required checks pass; the current self-hosted advisory failure is documented there and is not merge-required.

@michaelroy-amd
michaelroy-amd added this pull request to the merge queue Aug 17, 2026
Merged via the queue into main with commit 46f36d8 Aug 18, 2026
24 of 26 checks passed
@michaelroy-amd
michaelroy-amd deleted the fix-lemonade-missing-xdg-runtime-dir branch August 18, 2026 00:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rocm serve --engine lemonade fails on any headless host where XDG_RUNTIME_DIR is unset

2 participants