Repository navigation
fix(lemonade): give the server a runtime dir when XDG_RUNTIME_DIR is unset - #262
Conversation
…unset `rocm serve --engine lemonade --managed` failed deterministically on any Linux host without XDG_RUNTIME_DIR: the spawned server exits during startup with "Unable to resolve writable runtime directory from XDG_RUNTIME_DIR or RUNTIME_DIRECTORY". XDG_RUNTIME_DIR is populated by pam_systemd at login, so it is absent for every non-login process (cron jobs, CI runners, systemd-run, a bare container exec), and RUNTIME_DIRECTORY only exists for a systemd unit declaring RuntimeDirectory=. rocm-cli assembles the child environment, so it owns the seam. Reuse the runtime-directory precedence the dashboard socket already uses (XDG_RUNTIME_DIR, then $HOME/.rocm/data, then temp_dir()/rocm-<user>) by extracting its tier chain into rocm-core as `user_runtime_dir`, and give the engine a `lemonade` leaf inside the fallback tiers so a crashing server cannot disturb the telemetry socket sharing the same root. The directory is created mode 0700 before the server is spawned, and a symlink there is refused rather than chmod'ed through. A value already present in the parent environment is passed through unchanged. The readiness poll re-spawns a status probe twice a second, so it now resolves the child environment once instead of repeating the directory preparation on every attempt. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Signed-off-by: Michael Roy <michael.roy@amd.com>
…unset `rocm serve --engine lemonade --managed` failed deterministically on any Linux host without XDG_RUNTIME_DIR: the spawned server exits during startup with "Unable to resolve writable runtime directory from XDG_RUNTIME_DIR or RUNTIME_DIRECTORY". XDG_RUNTIME_DIR is populated by pam_systemd at login, so it is absent for every non-login process (cron jobs, CI runners, systemd-run, a bare container exec), and RUNTIME_DIRECTORY only exists for a systemd unit declaring RuntimeDirectory=. rocm-cli assembles the child environment, so it owns the seam. Reuse the runtime-directory precedence the dashboard socket already uses (XDG_RUNTIME_DIR, then $HOME/.rocm/data, then temp_dir()/rocm-<user>) by extracting its tier chain into rocm-core as `user_runtime_dir`, and give the engine a `lemonade` leaf inside the fallback tiers so a crashing server cannot disturb the telemetry socket sharing the same root. The directory is created mode 0700 before the server is spawned, and a symlink there is refused rather than chmod'ed through. A value already present in the parent environment is passed through unchanged. The readiness poll re-spawns a status probe twice a second, so it now resolves the child environment once instead of repeating the directory preparation on every attempt. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
7aa33bf to
afd657b
Compare
|
Rebased onto current
The one remaining red check, Identical to the baseline on Everything else — all 16 required checks, plus the Strix Halo Ubuntu lane — is green. |
|
Rebased onto The one red check,
The Windows lane fails on both, and on a different symptom: Re-verified locally after the rebase: |
Signed-off-by: Michael Roy <michael.roy@amd.com>
…f6cd Signed-off-by: Michael Roy <michael.roy@amd.com>
michaelroy-amd
left a comment
There was a problem hiding this comment.
- Review focus: Please review this as filesystem security code: fallback precedence, descriptor-relative tier-3 creation, ownership/mode checks, and unchanged operator-provided
XDG_RUNTIME_DIRbehavior. - Problem / impact: On headless Linux processes without
XDG_RUNTIME_DIRorRUNTIME_DIRECTORY, managed Lemonade exits before serving. CI runners, cron jobs, systemd-run invocations, and bare container execs therefore cannot use Lemonade. A naive/tmpfallback would add symlink, ownership, and path-replacement races. - State: #262, head
1f2ce58182f56b24e58fba6694733a5ef5b24bd4;OPEN, non-draft,MERGEABLE,REVIEW_REQUIRED(BLOCKED); 6 files, +1,172/-41. - Changed areas / review hotspots:
crates/rocm-core/src/runtime.rsandlib.rs: shared precedenceXDG_RUNTIME_DIR→$HOME/.rocm/data/<leaf>→<temp>/rocm-<sanitized-user>/<leaf>.crates/rocm-dash-core/src/config.rs: dashboard socket precedence remains behaviorally aligned without adding a heavy dependency.engines/lemonade/src/lib.rs: Linux-only childXDG_RUNTIME_DIR,lemonadeisolation leaf, and one-time environment resolution reused by readiness probes.- Tier-3 hardening:
openat/mkdirat,O_NOFOLLOW, descriptor-relative creation, effective-UID ownership, mode 0700, and ancestry checks that prevent cross-UID entry replacement. - Race/security regressions: symlinked leaves, foreign ownership, unsafe ancestors, temp-root replacement, user-parent replacement, and operator pass-through.
- Invariants to verify: A non-empty parent
XDG_RUNTIME_DIRpasses through byte-for-byte with no fallback mutation; HOME and temp fallbacks end in a privatelemonadeleaf; user input cannot escape its path component; every created/opened component is no-follow, correctly owned, and private; creation stays anchored to the validated directory object if pathnames are swapped; failures are hard errors, never a permissive fallback; Windows behavior and strict GPU/no-CPU-fallback policy are unchanged. - Verification evidence: Required: all required hosted checks green, including Linux/Windows builds/tests, affected tests, clippy, signature+DCO, hygiene, and CodeQL. Advisory: GPU, Strix Ubuntu, and the self-hosted consolidated report pass—this is the only packet whose GPU leaf is green. Red: Strix Windows has five unexpected failures (
bench-load-real-serve,chat-tool-definitions-accepted,chat-end-to-end-local-model,serve-lemonade-inference,serve-hf-checkpoint-inference) after the llama.cpp backend download exhausts six attempts withSSL connect error (CURL code: 35). - Recommended sequence: shared resolver → Lemonade child-env call sites → fd-relative hardening → adversarial tests → dashboard parity/platform gates.
michaelroy-amd
left a comment
There was a problem hiding this comment.
Approved at 1f2ce58182f56b24e58fba6694733a5ef5b24bd4. The focused review above covers the changed contracts and invariants. All required checks pass; the current self-hosted advisory failure is documented there and is not merge-required.
Fixes #259.
Symptom
rocm serve --engine lemonade --managedfails on any Linux host with noXDG_RUNTIME_DIR. It is fully deterministic and independent of the model or the servepath:
The same command works from an interactive desktop or SSH login on the same machine, so
it reads as a machine quirk when it is really a property of how the process was started:
a systemd unit, a cron job, a container,
ssh host 'rocm serve ...'without lingering,or any CI runner.
Root cause
The message comes from the spawned server, not from rocm-cli — the string appears nowhere
in this tree and
RUNTIME_DIRECTORYis never referenced here. The server needs a writableruntime directory and consults exactly two sources, neither of which exists in a headless
context:
XDG_RUNTIME_DIR, populated bypam_systemdat login and therefore absent for everynon-login process;
RUNTIME_DIRECTORY, which only exists for a systemd unit declaringRuntimeDirectory=.lemonade_process_environment_varsis the single place the child environment isassembled. It translated
ROCM_PATH,PATH,LD_LIBRARY_PATH,HIP_VISIBLE_DEVICESandLEMONADE_API_KEY, but nothing about the runtime directory. rocm-cli cannot change whatthe server requires, but it owns the environment it hands over, so this is the layer that
can satisfy it.
Fix
Reuse the precedence this repo already has rather than inventing a second one.
dashboard_socket_pathalready solves "pick a writable, user-owned runtime directory witha fallback chain", as a pure function taking its env inputs as arguments:
$XDG_RUNTIME_DIR$HOME/.rocm/data/…temp_dir()/rocm-<user>, with the user name sanitized so a separator or..cannotescape the subdirectory
That tier chain moves into
rocm_core::user_runtime_dir;dashboard_socket_pathbecomesa thin caller over it and resolves to byte-identical paths in all three tiers (its three
existing tests are unchanged and still pass). The engine calls the same helper when the
parent environment has no usable value, creates the directory mode
0700before spawning,and passes it down.
Why not fix this in CI
A workflow-level fix was considered and rejected. Exporting
XDG_RUNTIME_DIRin.github/workflows/e2e-selfhosted.yml, or setting it for piped scenarios in the E2Eharness, would turn the GPU lane green and leave every headless user exactly as broken.
pty_env(tests/e2e-cucumber/tests/e2e.rs) does setHOMEandXDG_RUNTIME_DIR, butonly for PTY scenarios — its comment records that piped scenarios keep their historical
HOME/XDG environment, and the serve scenarios are piped. That is correct and stays: it is
precisely why the suite reproduces the real headless case instead of hiding it.
Decisions worth reviewing
A distinct directory for the engine, not the dashboard's. In the fallback tiers the
engine gets a
lemonadeleaf ($HOME/.rocm/data/lemonade,temp_dir()/rocm-<user>/lemonade). Sharing one directory would put engine scratch statenext to the telemetry socket, so a crashing or misbehaving server could disturb the
dashboard. The isolation costs one path component.
A shared helper, but only for the two callers that can share one. The tier chain is
extracted into
rocm-core, used bydashboard_socket_pathand by the engine, so thereare still two copies of this logic rather than three.
rocm-dash-corekeeps its ownmirror deliberately: it is a standalone library with a lean dependency set, and depending
on
rocm-coreto share ~30 lines would pull in that crate's whole graph. Its "keep thetwo in sync" comment now names the shared helper so the sync target is unambiguous, and
the mirrored tests on both sides still catch divergence.
XDG_RUNTIME_DIRrather thanRUNTIME_DIRECTORY. The server reads either.RUNTIME_DIRECTORYis narrower, but it is a systemd-unit contract implying systemdcreated the directory and will clean it up — untrue here, and misleading to anyone
debugging.
XDG_RUNTIME_DIRis the freedesktop-standard variable whose absence is the bug;supplying a valid one is strictly better for any descendant process than leaving it unset.
It is never overridden: a value already in the parent environment is passed through
unchanged.
Hardening. The tier-3 fallback lands under a shared temp dir, so a pre-existing path
there may not be ours. A symlink is refused rather than
chmod-ed through, and a chmodfailure is a hard error naming the path.
Verification
Ran locally on a headless Linux host (no
XDG_RUNTIME_DIR, noRUNTIME_DIRECTORY):cargo fmt --all --checkcargo clippy --locked --workspace --all-targets -- -D warningscargo clippy --locked -p e2e-cucumber --test e2e -- -D warningscargo test --workspace --all-targets, plus the existingdashboard_socket_pathandrocm-dash-coremirror tests$HOME/.rocm/data/lemonade, createddrwx------Five new unit tests on
lemonade_process_environment_varscover synthesis when the parenthas no value, an exported-but-empty value, the no-
HOMEtier, pass-through of an existingvalue, and refusal of a symlinked directory. All five fail before this change and pass
after. They take the environment as arguments rather than mutating process-global env, so
they behave identically under
cargo testand nextest.Re-verified after rebasing onto current
main: all five gates above pass, andcargo test --workspace --all-targetsis now fully green. Two shared-processcargo testfailures noted here earlier
(
therock::tests::extracting_the_sdk_archive_removes_it,providers::tests::local_provider_default_chat_requires_builtin_qwen_assistant) werepre-existing on
mainand are gone since #253 landed; nothing about them was ever relatedto this change.
The real verification is the GPU lane, and it is now green. No GPU here, so a full
managed serve could not be run locally;
@id:serve-lemonade-inferenceand@id:serve-hf-checkpoint-inference(@requires-gpu @requires-engine:lemonade) exercisethis on
E2E tests (GPU), the MI300X lane where the failure reproduces.serve-hf-checkpoint-inferenceserve-lemonade-inferencestill xfails, but on the unrelated EAI-7423 symptom it isalready registered for ("did not serve model within 90s"), and it now gets past server
startup instead of exiting immediately.
Worth being explicit about the shape of that coverage: the scenarios are piped, so they
inherit the runner's environment rather than asserting on it — they regress this bug
because that runner is headless, not because the harness forces it. Making it explicit
would mean touching the piped scenarios' environment, which is the thing this fix
deliberately avoids.
tests/e2e-cucumber/expectations.tomlwas searched: no row describes this failure mode.The lemonade Linux rows there are EAI-7423 ("reaches ready then shuts down immediately") on
the Strix Halo Ubuntu lane, a different symptom on a different host, and are left alone.