Skip to content

fix: raise the supported Linux minimum to Ubuntu 24.04 - #261

Merged
rominf merged 2 commits into
mainfrom
docs/ubuntu-2404-minimum
Aug 27, 2026
Merged

rominf merged 2 commits into
mainfrom
docs/ubuntu-2404-minimum

Conversation

@rominf

@rominf rominf commented Aug 14, 2026 •

Copy link
Copy Markdown
Collaborator

The problem

The docs promise a platform one of the two shipped engines cannot run on.

README.md advertised Linux (x86_64) as "Full support, including the live
dashboard and both inference engines"
, and docs/wsl.md listed
"WSL2 with Ubuntu 24.04 or Ubuntu 22.04" as a prerequisite. Together those tell a
22.04 user — native or WSL2 — that both engines work.

They do not. Every published Lemonade embeddable, v10.2.0 through v11.5.2 (all 15
ubuntu-x64 releases), is linked against GLIBC_2.38 and GLIBCXX_3.4.32. Ubuntu
22.04 ships glibc 2.35 and GLIBCXX_3.4.30, so lemond is refused by the dynamic
loader before it starts. There is no older release to pin back to and no supported
way to put glibc 2.38 on jammy. Full analysis in #258.

This is option 1 from that issue — narrow the documented matrix to match reality.

What changed

One flat minimum: Ubuntu 24.04 or newer, for Linux and WSL2 alike.

  • README.md: both the Linux and WSL2 rows now say "Ubuntu 24.04 or newer", and a
    short paragraph below gives the glibc number behind that line — glibc 2.38 with
    GLIBCXX_3.4.32 — so the requirement still means something to a reader on RHEL,
    SLES, or Fedora who cannot map an Ubuntu version onto their own distribution.
  • docs/wsl.md: prerequisite 2 is now "Ubuntu 24.04 or newer", with the same reason
    stated inline.
  • scripts/wsl_preflight.py: 22.04 dropped from the supported set, and the
    ubuntu check detail now names the minimum. Covered by a new --self-test case
    (an otherwise-ready 22.04 host must fail the check), which fails against the old
    set and passes against the new one, and runs in CI on both the Linux and Windows
    lanes.

The minimum is deliberately stated as a single number rather than split per engine.
The Lemonade floor is the binding one; a per-engine matrix would be more precise but
harder to state and harder to test, and the simpler line is the one being supported.

The one edit a maintainer might want reverted

scripts/wsl_preflight.py is a behaviour change, not documentation — hence the fix:
title rather than docs:. It is included because that script is the automated form of
the docs/wsl.md prerequisite list and is invoked directly from it; leaving it
reporting ubuntu: ok on a host Lemonade cannot run on would relocate the
inconsistency rather than fix it. If you would rather keep this PR strictly to prose,
it is a self-contained three-line revert.

Also found, deliberately not changed

  • apps/rocm/src/main.rs driver-install matrix (("ubuntu", "22.04" | "24.04"),
    codename_for_version). That mirrors AMD's own documented amdgpu-dkms install
    matrix, which is a separate upstream support matrix; rocm install driver still
    produces a correct plan on 22.04 and narrowing it would be a behaviour change beyond
    this issue.
  • crates/rocm-core/src/diagnose.rs suggesting rocm/dev-ubuntu-22.04 as a
    known-good container image. A container brings its own userland, so the host glibc
    claim does not apply; it is not a host-OS support statement.

This does not fix the user experience, only the promise

#258 also asks for a preflight error — "the lemonade engine requires glibc 2.38 or
newer; this host has 2.35"
— instead of a raw loader message. That is deliberately
out of scope here, but worth stating plainly: no such preflight exists today. The
only glibc-aware code in the tree is a low-weight "version `?glibc" keyword in the
rocm diagnose classifier, explicitly annotated "tangential"; there is no glibc check
anywhere in the Lemonade download or launch path. A 22.04 user who has not read these
docs still gets an opaque version `GLIBC_2.38' not found out of a subprocess. The
preflight remains worth doing on its own, and it would be worth tracking separately
rather than letting it lapse when this merges.

Related

Verification

prek run --all-files --no-group local-tools (all passed: whitespace, EOF, yaml,
merge-conflict, line endings, ruff check, ruff format, shellcheck, cargo fmt),
hawkeye check --config licenserc.toml (no missing headers), cargo fmt --all --check (clean), python3 scripts/wsl_preflight.py --self-test (ok; and confirmed
the new assertion still fails against the pre-change supported set).

No Gherkin scenario accompanies this: the rocm/rocmd binaries have no observable
behaviour change. The one behavioural edit is to a standalone helper script, whose
contract is covered by its own --self-test, already run by CI.

Searched tests/e2e-cucumber/expectations.toml for stale xfails tied to this issue —
the Lemonade rows there are all EAI-7423 (managed serve shutting down after ready),
unrelated to the glibc floor, so nothing to remove or narrow.

Fixes #258

@rominf
rominf requested a review from a team as a code owner August 14, 2026 12:15
@rominf

rominf commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

CI status, so the one red check is not left unexplained.

All 16 required checks are green, including prek (lint / hygiene), License header check (hawkeye), Commit signatures + sign-off, E2E tests, and both CodeQL analyses.

E2E tests (GPU) is red, and it is pre-existing on main, not caused by this PR. The lane is advisory (not in the required set). Evidence:

  • The failure is one unexpected failure, serve-hf-checkpoint-inference, failing at "a canonical Hugging Face GGUF checkpoint is being served on lemonade" with rocm serve failed. Reconciliation line: 5 xfail, 0 XPASS, 1 unexpected failure(s).
  • The main-branch run from the same window (job 94765233909) fails with the identical reconciliation line and the identical scenario, so the regression is already on main.
  • This PR changes README.md, docs/wsl.md, and scripts/wsl_preflight.py. None of the three is read by the rocm binary or the E2E harness, and the GPU runner is Ubuntu 24.04.4 LTS — above the new documented minimum either way, so nothing in this change can reach that lane.
  • The symptom is a lemonade serve failure on a headless self-hosted runner, which is what rocm serve --engine lemonade fails on any headless host where XDG_RUNTIME_DIR is unset #259 describes: the runner's glibc 2.39 clears the floor tracked in Lemonade engine cannot start on Ubuntu 22.04 (glibc 2.35), but README and docs/wsl.md advertise it as supported #258, so the XDG_RUNTIME_DIR failure underneath is no longer masked. There is already a fix-lemonade-missing-xdg-runtime-dir branch in flight for it.

E2E tests (Strix Halo, Ubuntu) passed. E2E tests (Strix Halo, Windows) is still queued behind the self-hosted runner backlog; it is also advisory and I will follow up if it reports anything other than the same pre-existing state.

@rominf

rominf commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up on the two advisory self-hosted lanes, now that both have finished.

  • E2E tests (Strix Halo, Ubuntu) — passed.
  • E2E tests (Strix Halo, Windows) — failed, and it is the same pre-existing failure: 3 xfail, 0 XPASS, 1 unexpected failure(s) / serve-hf-checkpoint-inference. The unrelated branch feat/rocm-version-tag-branch-hash (job 94760021263), which touches no docs, produces a byte-identical reconciliation line and the same scenario, so this is repo-wide rather than anything in this PR. main's own Windows lane is still queued behind the runner backlog; its GPU lane already failed the same way (job 94765233909).

So both red lanes fail on serve-hf-checkpoint-inference, on hosts already above the new documented minimum, and both reproduce off this branch. Neither is in the 16 required checks, and all 16 required checks are green.

@rominf
rominf force-pushed the docs/ubuntu-2404-minimum branch from ceb0d59 to 7b66483 Compare August 14, 2026 13:47
@rominf rominf changed the title docs: raise the documented Linux minimum to Ubuntu 24.04 fix: raise the supported Linux minimum to Ubuntu 24.04 Aug 14, 2026
@rominf

rominf commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Updated per maintainer decision: one flat minimum of Ubuntu 24.04, no per-engine matrix. The glibc number stays as the reason behind that line so it still means something on a non-Ubuntu distribution. Retitled docs: → fix:, since the diff includes the scripts/wsl_preflight.py behaviour change.

Head is now 7b66483, rebased onto current main (0063ef15).

CI on the new head: all 16 required checks green. Re-verified locally after the edit — prek run --all-files --no-group local-tools, hawkeye check, cargo fmt --all --check, and wsl_preflight.py --self-test including the new 22.04 regression case (confirmed it still fails against the pre-change supported set).

E2E tests (GPU) is red again for the same pre-existing serve-hf-checkpoint-inference / #259 reason explained above; the two Strix lanes are still running. All three are advisory, none is in the required set, and none is reachable from a change to two markdown files and a helper script.

@r0x0r r0x0r left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff and the linked analysis in #258. This is a clean, well-scoped change and the reasoning holds up. Verified locally by running the PR's python3 scripts/wsl_preflight.py --self-test — it passes, and the new cases exercise the numeric comparison correctly (accepts 24.10/25.04/26.04/28.04, rejects 22.04, and fails closed on 24.04.1, unknown, empty, and missing VERSION_ID).

What I like

  • The wsl_preflight.py change moves from a hardcoded version set to a parsed (major, minor) tuple compared against (24, 4). That's a real improvement: "or newer" now actually works for future releases instead of needing the set extended each cycle, and the parse-fails-closed behavior is the right default for a preflight.
  • Docs and the automated check are kept in lockstep, and the glibc floor (2.38 / GLIBCXX_3.4.32) is stated as the distro-neutral reason rather than just an Ubuntu version number — useful for RHEL/SLES/Fedora readers.
  • The "deliberately not changed" section (driver-install matrix in main.rs, the dev-ubuntu-22.04 container hint in diagnose.rs) matches what I see in the tree; leaving those is correct since they're a separate support matrix and a container userland respectively.

Minor nits (non-blocking)

  • The tuple floor (24, 4) represents 24.04 with minor 4. It reads fine given the inline comment, but a reader skimming might expect 04. Not worth changing.
  • Strictly on the glibc floor, Ubuntu 23.10 shipped glibc 2.38 and would technically clear the Lemonade requirement, yet the preflight rejects it. That's a defensible simplification (23.10 is non-LTS and EOL, and the PR argues explicitly for a single simple LTS line), just noting the small precision gap.
  • Fail-closed on 24.04.1: real Ubuntu os-release VERSION_ID is 24.04, not a point release, so this won't bite in practice — good to have it locked down by the self-test regardless.

Scope note (already disclosed, agreeing)

The PR is honest that it fixes the promise, not the UX — a 22.04 user still gets the raw loader error with no glibc preflight. Agree that's out of scope here and worth tracking separately so it doesn't lapse.

Nothing blocking from my side.

@rominf

rominf commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the review. On the three minor nits — agreed with your own calls, no changes: the (24, 4) tuple reads fine with the inline comment, the 23.10 gap is an accepted simplification for a single LTS line, and the 24.04.1 fail-closed case is intentional belt-and-suspenders since real os-release never reports a point release there.

On the scope note: opened #289 to carry the preflight-error ask forward on its own, since this PR is titled Fixes #258 and merging it would otherwise close that issue with the preflight half of the ask still undone.

rominf and others added 2 commits August 24, 2026 16:13
Every published Lemonade embeddable, v10.2.0 through v11.5.2, is linked
against GLIBC_2.38 and GLIBCXX_3.4.32. Ubuntu 22.04 ships glibc 2.35, so
the Lemonade engine cannot start there at all, yet README.md advertised
Linux x86_64 as "full support ... both inference engines" and docs/wsl.md
named 22.04 as a supported WSL base.

Document a single minimum of Ubuntu 24.04 for Linux and WSL2, keeping the
glibc number as the reason so the requirement stays meaningful on other
distributions, and drop 22.04 from the WSL preflight's supported set so the
automated check matches the prerequisite it enforces.

Fixes #258

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Signed-off-by: Michael Roy <michael.roy@amd.com>
@r0x0r
r0x0r force-pushed the docs/ubuntu-2404-minimum branch from fe11f17 to 803f00f Compare August 24, 2026 13:14
@rominf
rominf requested a review from r0x0r August 26, 2026 10:32
@rominf

rominf commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator Author

Following up — the review notes were addressed (replied inline above), and this needs a formal approving review to satisfy the code-owner requirement (a comment-only review doesn't clear it). Re-requesting review.

@michaelroy-amd michaelroy-amd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 803f00ff394b5a9e41e7c9940268eebff7312e2a. The documentation and wsl_preflight.py now enforce one Ubuntu 24.04 minimum that matches Lemonade's glibc requirement. The version parser fails closed, and the self-test covers supported, unsupported, malformed, and missing VERSION_ID values. Driver-install and container support matrices remain correctly out of scope. All 17 required checks are green, both commits are signed off, and I found no blocking issue. Approving.

@rominf
rominf added this pull request to the merge queue Aug 27, 2026
Merged via the queue into main with commit 7982de1 Aug 27, 2026
25 checks passed
@rominf
rominf deleted the docs/ubuntu-2404-minimum branch August 27, 2026 08:36
pmoutsias-amd pushed a commit that referenced this pull request Aug 27, 2026
* fix: raise the supported Linux minimum to Ubuntu 24.04

Every published Lemonade embeddable, v10.2.0 through v11.5.2, is linked
against GLIBC_2.38 and GLIBCXX_3.4.32. Ubuntu 22.04 ships glibc 2.35, so
the Lemonade engine cannot start there at all, yet README.md advertised
Linux x86_64 as "full support ... both inference engines" and docs/wsl.md
named 22.04 as a supported WSL base.

Document a single minimum of Ubuntu 24.04 for Linux and WSL2, keeping the
glibc number as the reason so the requirement stays meaningful on other
distributions, and drop 22.04 from the WSL preflight's supported set so the
automated check matches the prerequisite it enforces.

Fixes #258

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(wsl): accept Ubuntu releases after 24.04

Signed-off-by: Michael Roy <michael.roy@amd.com>

---------

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Signed-off-by: Michael Roy <michael.roy@amd.com>
Co-authored-by: Michael Roy <michael.roy@amd.com>
Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>
mikeroySoft pushed a commit to mikeroySoft/rocm-cli that referenced this pull request Sep 1, 2026
* Add Sphinx documentation site

Adds a Sphinx documentation site under docs/rocm-docs/, single-sourced
from README.md and CONTRIBUTING.md via MyST {include} directives so
the docs and repo content stay in sync automatically. Covers
installation, getting started, command reference, demos, and
contributing/license, and is set up to publish with the rocm-docs-core
theme. Updates some links in README.md and CONTRIBUTING.md to be
absolute so they render correctly when reused via {include}.

Signed-off-by: pmoutsia_amdeng <peter.moutsias@amd.com>

* Single-source the platform-support table in the install page

Anchor the table in README.md and pull it into
docs/rocm-docs/install/installation.md via {include} instead of
duplicating it by hand.

Signed-off-by: pmoutsia_amdeng <peter.moutsias@amd.com>

* fix(docs): address review feedback on Sphinx docs site

- Add the MIT license header to the 7 new files hawkeye flagged as
  missing it.
- Fix the WSL2 platform-support-table link to docs/wsl.md: it's now an
  absolute GitHub URL so it resolves when the table is pulled into
  install/installation.md via MyST include, matching the pattern
  already used for the other README anchors.
- Pin rocm-docs-core to the exact version that resolved locally
  (1.40.0) instead of an open->= range.

Signed-off-by: pmoutsia_amdeng <peter.moutsias@amd.com>

* fix(docs): add ReadTheDocs config and a CI job to build docs with -W

- Add .readthedocs.yaml so RTD builds against docs/rocm-docs/conf.py
  with the same pinned rocm-docs-core toolchain CI uses.
- Add a docs-build CI job that runs sphinx-build -W (warnings as
  errors) against the same source RTD points at, gated on the new
  `docs` path-filter category so it only runs when doc sources,
  README.md/CONTRIBUTING.md (single-sourced via MyST includes), or
  the build config change.
- Suppress the myst.header warning from :start-after:/:end-before:
  README includes stripping the anchor heading, which is cosmetic:
  docutils re-normalizes the resulting heading depth in the rendered
  output.
- Scope external_projects to [] instead of the implicit "all" default,
  which was fetching intersphinx inventories for every project in
  rocm_docs' bundled catalog (~90 projects) - several of which 404 or
  have moved upstream. No page here uses a cross-project intersphinx
  role, so this is lossless and makes -W builds deterministic instead
  of flaky against infrastructure this repo doesn't control.
- Filter the "current project not found in projects" warning: rocm-cli
  isn't registered in rocm-docs-core's shared projects catalog yet, so
  external_projects_current_project can never resolve until that's
  added upstream. rocm_docs.projects handles the unresolved project as
  None everywhere it's used, so this is informational, not a defect.

Signed-off-by: pmoutsia_amdeng <peter.moutsias@amd.com>

* docs: use docs-relative cross-page links consistently

README.md's own #model-serving and #interactive-interfaces anchors were
rewritten to absolute GitHub URLs so they'd still work once pulled into
the docs site, but that breaks in-page anchor scrolling when the README
is viewed on GitHub. Revert both to relative anchors and instead apply
the cut-and-reauthor pattern already used for the CONTRIBUTING.md link
in installation.md: narrow the surrounding {include} in getting-started.md
and commands.md around each sentence and hand-author a docs-relative
replacement (commands.md#model-serving, getting-started.md#interactive-interfaces).

Signed-off-by: pmoutsia_amdeng <peter.moutsias@amd.com>

* docs: use descriptive nav title for Demos section

Avoids the sidebar rendering as duplicated Demos > Demos for the
single-entry section by overriding the nav link text to match the
homepage tile's wording.

Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* docs: fix sentence fragments in intro blurb

Joins the "single prebuilt binary" fragments into one complete
sentence in both README.md and the standalone index.rst homepage
copy, and adds a lead-in sentence before the bare `rocm` code fence
in the Getting Started "First run" section.

Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* docs: style guide pass on README and CONTRIBUTING

Sentence-case headings, expand DCO acronym, replace banned/informal
wording (may, e.g.), normalize slash usage, lowercase version
placeholder.

Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* fix(lemonade): retry interrupted backend setup (ROCm#249)

* fix(lemonade): retry interrupted backend setup

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>

* ci(e2e): build prebuilt lanes with the test-hooks feature

The suite's deterministic failure seams are compiled out without
`rocm/e2e-test-hooks`. `cargo xtask e2e` passes the feature only when it
builds the binaries itself, so every lane that pre-builds and exports
ROCM_CLI_BINARY silently tested a binary without those seams. The
scripted-Lemonade-failure scenario then never reached its premise and
failed as a regression on the one lane that selects it.

Pass the feature on all prebuilt e2e lanes, and add a workflow-contract
test so a lane cannot drift back. Release and packaging builds keep the
feature off — a shipped binary must not carry a failure-injection seam.

Also name the cause in the retry assertion, which otherwise reports a
baffling missing announcement rather than the real misconfiguration.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>

---------

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* ci(dependabot): track pre-commit hook versions (ROCm#235)

The hook pins in .pre-commit-config.yaml were bumped by hand and drifted
between releases. Dependabot's pre-commit ecosystem resolves each rev
against the hook repository's tags, and skips the builtin and local
blocks, which have no upstream release to track.

Weekly and grouped, with the same 7-day cooldown as the other two
ecosystems: a hook is executable code that runs on every contributor's
machine at commit time and in the prek CI job, so a compromised release
would run before anyone reads the bump.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* fix: raise the supported Linux minimum to Ubuntu 24.04 (ROCm#261)

* fix: raise the supported Linux minimum to Ubuntu 24.04

Every published Lemonade embeddable, v10.2.0 through v11.5.2, is linked
against GLIBC_2.38 and GLIBCXX_3.4.32. Ubuntu 22.04 ships glibc 2.35, so
the Lemonade engine cannot start there at all, yet README.md advertised
Linux x86_64 as "full support ... both inference engines" and docs/wsl.md
named 22.04 as a supported WSL base.

Document a single minimum of Ubuntu 24.04 for Linux and WSL2, keeping the
glibc number as the reason so the requirement stays meaningful on other
distributions, and drop 22.04 from the WSL preflight's supported set so the
automated check matches the prerequisite it enforces.

Fixes ROCm#258

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(wsl): accept Ubuntu releases after 24.04

Signed-off-by: Michael Roy <michael.roy@amd.com>

---------

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Signed-off-by: Michael Roy <michael.roy@amd.com>
Co-authored-by: Michael Roy <michael.roy@amd.com>
Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* ci(e2e): repair a pre-warm tree that records a folder elsewhere (ROCm#316)

A scenario reaches the shared pre-warm tree through a symlink at its own
`data/runtimes`. An `install sdk` run that way writes the link's path — a
per-scenario temp dir — into the manifest that lands in the SHARED
registry, and into the venv's console-script shebangs. The temp dir goes
away with the scenario and the shared runtime is left naming a folder
that no longer exists, so later, unrelated runs fail.

Drop such a runtime before deciding whether the tree is fresh, so the
pre-warm reinstalls instead of serving a dead one. `rocm update` cannot
see this: it compares versions against the index, not the runtime
against the disk.

Removing the folder is the repair, not a precaution. A poisoned venv
keeps a working `bin/python` — a symlink to the base interpreter, still
present — so the install reuses it and audits already-satisfied packages
rather than reinstalling them, leaving every shebang pointing at the
folder that went away. Measured on uv 0.9.30: reinstalling over a
poisoned venv reports success and repairs nothing.

The signal is an install root outside the tree, deliberately not
`status=unusable`. Unusable has many causes — a missing rocm_sdk probe
block alone reports it — and this deletes what it selects, so a healthy
multi-GiB runtime must not hang on a validation detail. Read-only
runtimes are exempt: `runtimes adopt` records an external folder on
purpose.

Refs: ROCm#315

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* fix(install): record the folder the files land in, not the link (ROCm#317)

* fix(install): record the folder the files land in, not the link

`install sdk` built the install root by joining onto the data dir and
recorded whatever that produced. Reaching `data/runtimes` through a
symlink is enough to make that name a route rather than a place: correct
while the link exists, and wrong the moment it goes.

The path outlives the command. It goes into the registry manifest, the
sidecar beside the runtime, and — through uv — the `#!` line of every
console script in the environment. So a runtime installed that way keeps
reporting itself installed at a folder that is not there, while the files
sit untouched next door and every entry point fails to start.

Resolve the root before anything is written to it. Only the root:
`python_executable` is derived from it and must keep the venv's own
`bin/python`, which is itself a symlink to the base interpreter, so
resolving that would record the system Python instead. The adopt path
already draws the line in the same place, canonicalizing the install root
next to `absolute_existing_file_path_preserving_symlink` for the
interpreter; this gives the install path the same treatment, `--prefix`
included.

The E2E harness creates exactly this shape when a scenario opts into the
shared pre-warmed runtime, so the shared tree on a runner was what got
poisoned — but nothing about the fault is test-only. Any data dir reached
through a link has it.

Refs: ROCm#315
Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>

* fix(vllm): name the missing interpreter behind a failed launch

Two messages described a stale runtime as an absence.

A spawn that fails on a console script whose `#!` interpreter is gone
reports ENOENT against the script, so the error named a file that is
plainly there. Say which interpreter is missing when that is what
happened, and stay quiet when the ordinary reading is right — a genuinely
absent file, a binary, or a live interpreter.

Resolution drops a registered runtime whose recorded interpreter is not
there. That is correct — a runtime that cannot run is not a candidate —
but it left the failure describing an empty registry while `rocm runtimes
list` prints the entry. Name the manifests that were passed over and the
interpreters they record.

Both are additive: the note only ever decorates an error already being
returned, and a registry it cannot read yields no note rather than
replacing the original failure.

First tests for this area, which had none.

Refs: ROCm#315
Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>

* test(install): compare resolved paths, not verbatim ones, on Windows

The Windows lane failed on the prefix rather than the behaviour.
`canonicalize` hands back `\\?\C:\…` there, so a test that built its
expectation that way compared a verbatim path against the plain one the
resolver deliberately returns, and three assertions turned on the
difference.

Build the expectations with the resolver instead, which strips the prefix
for the same reason the product does: a stored path is later compared
against ordinary ones, and `\\?\C:\…` never `starts_with`-matches `C:\…`.
That the prefix really is stripped stays asserted on its own, so this does
not become the function agreeing with itself.

This also removes a second Windows-only trap: `temp_dir()` there can
return an 8.3 short path, which `canonicalize` expands, so the two sides
disagreed on the folder name as well as the prefix.

Same fault in the scenario's step definitions, where the comparison would
have failed only on the Strix Windows lane. That crate cannot reach
`rocm-core`, and a dependency for six lines of string handling is the
worse trade, so it strips the prefix locally with the reason recorded.

Refs: ROCm#315
Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>

* fix(install): give up on a parent component consistently across hosts

The guard against resolving a `..` past a missing directory never fired
on Windows, because the two platforms disagree about what such a path
means. Windows collapses `..` lexically, so `<root>/missing/..` "exists"
and canonicalizes to `<root>`; Unix walks the path through the filesystem
and finds nothing. The walk therefore stopped a level early on Windows,
resolved to the wrong folder, and re-attached the tail — recording a
different folder depending on the host, which is precisely what this
function exists to prevent.

Check for the parent component before asking whether the candidate
exists, so both hosts give up in the same place.

The give-up stays narrow: a `..` whose parent is really there is
unambiguous everywhere and still resolves, which is now pinned by its own
test so a future tightening cannot quietly leave ordinary paths
unresolved.

Found by the Windows lane; it does not reproduce on Linux.

Refs: ROCm#315
Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>

---------

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* fix(install): resolve driver repo version and package release in plans (ROCm#307)

* fix(install): resolve driver repo version and package release in plans

The driver install plan stored the repo version and package release as
raw shell parameter-expansion templates (${ROCM_CLI_AMDGPU_VERSION:-7.2.4}
and ${ROCM_CLI_AMDGPU_PACKAGE_RELEASE:-70204}). Two problems followed.
The human-readable plan summary printed the version template verbatim on
its "repo_version:" line, leaking an unexpanded shell placeholder into
user-facing dry-run output. And the Debian/Ubuntu apt source line wrapped
the template in single quotes, so the shell never expanded it and the
literal placeholder would have been written into
/etc/apt/sources.list.d/amdgpu.list.

Resolve both templates once, at plan-build time, to their effective
values (the env var when set and non-empty, matching shell :- default
semantics, otherwise the documented default) and bake the concrete values
into every rendered URL, repo path, and command. The resolver is
generalized to resolve_shell_default_template and hardened to pass bare
${VAR} and nested-default shapes through unchanged rather than emitting a
partially rewritten value.

Add unit tests for the resolver (default, env override, empty-as-unset,
non-template pass-through, bare-var and nested-default pass-through) and
update every distro plan test to assert the concrete values, plus a
render-level regression test and an end-to-end scenario asserting the
dry-run summary shows the resolved version rather than the raw
placeholder. Tests that read these process env vars serialize on a shared
lock and save/restore prior values so they stay deterministic under
edition-2024 env semantics.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(install): guard the two remaining env-reading driver-plan tests

build_driver_install_plan now resolves the repo-version template via
std::env::var at the top of the function, so every test calling it reads
process env. The sweep that added ScopedTestEnv::with_amd_overrides_cleared()
to the other call sites missed windows_install_driver_is_validate_only and
wsl_install_driver_uses_rocdxg_guidance_without_dkms, leaving two unguarded
readers racing the guarded mutators in the same test binary. Add the guard to
both. No production change.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

---------

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* fix(cli): accept negative float flag values in space form (EAI-8243) (ROCm#306)

* fix(cli): accept negative float flag values in space form

The space form of negative-number float flags (e.g. `serve --temperature
-1`) was rejected by clap with a confusing "unexpected argument '-1'"
error, because clap parsed the leading-dash token as a flag rather than a
value. Only the equals form (`--temperature=-1`) reached the range
validator that reports the valid range.

Enable `allow_negative_numbers` on the affected float flags
(`--temperature` and `--top-p` on both `chat` and `serve`) so the space
form reaches the value parser and both forms validate identically,
surfacing a clear range-validation message instead of an unexpected
argument error.

Add regression tests covering both the space and equals forms for
`--temperature` and `--top-p` on `chat` and `serve`.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(cli): extend negative-value fix to --max-tokens and add serve scenario

The negative-number-as-flag gotcha is a tokenizer-level ambiguity that
fires before any value parser runs, so it is not float-specific:
`--max-tokens -1` (space form) on both chat and serve was still rejected
as an unexpected argument while `--max-tokens=-1` reached parse_positive_u32.
Add allow_negative_numbers to --max-tokens on both commands so the space
form reaches the value parser and reports a clear error, matching the
--temperature/--top-p behavior fixed here.

Per AGENTS.md §3, the user-observable stderr change is now covered by an
ungated Gherkin scenario, @id:serve-negative-temperature-rejected, in
model_serving.feature. The value parser runs inside argument parsing
before engine selection or GPU pre-flight, so the scenario needs no GPU
and no engine and gates every PR (mirrors the ungated
@id:fix-position-argument-rejected precedent).

Also: reword the shared rationale comment to cover all three numeric value
flags (and note clap only treats a token as negative when the whole
remainder parses as one, so missing/malformed values still error at parse
time), give the chat regression test a symmetric comment, and extend both
space-form regression tests with the --max-tokens cases.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(e2e): number the negative-temperature serve scenario

The new scenario was the only unnumbered title in model_serving.feature;
every other scenario uses 'Scenario: <N> - <Title>'. Assign the next free
number (15) to keep the file consistent. No behavior change.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

---------

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* fix(serve): validate --device via clap possible values (exit 2) (ROCm#304)

* fix(serve): validate --device via clap possible values (exit 2)

Model `rocm serve --device` as a clap `ValueEnum` (`DevicePolicyArg`) so an
invalid value is rejected by clap's usage validation (exit code 2) with the
valid choices listed, instead of parsing as a free-form string and failing
later in application logic with a generic "unsupported device policy" error
(exit code 1).

This makes `--device` consistent with every other enum-style argument in the
CLI (`--engine`, config set-*, engines install, etc.), all of which use clap
possible values. The historical aliases (`auto`/`gpu` for gpu_required, `cpu`
for cpu_only) stay accepted for backward compatibility but are hidden from the
advertised list, and the intentional cpu_only app-level rejection (exit 1 with
a GPU-required message) is preserved.

Adds regression tests asserting invalid `--device` is a clap InvalidValue usage
error and that every value and alias still parses, and reworks the device
possible-values sync test to read clap's structural possible values rather than
a hand-written doc string.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(serve): correct --device help text and hide cpu_only choice

The per-variant ValueEnum doc comments render in `serve --help`, shell
completions, and the invalid-value suggestion list. `gpu_preferred`
previously read "Prefer a ROCm GPU when one is available", which is
misleading: parse_device_policy maps gpu_preferred to GpuRequired and
rocm serve has no CPU fallback path. Reword it to state it behaves
identically to gpu_required.

cpu_only is always rejected (rocm serve requires GPU execution), so hide
it from the advertised choices with #[value(hide = true)] while keeping it
parseable so the deliberate rejection message is preserved.

Add device_policy_arg_maps_through_parse_device_policy to guard the
DevicePolicyArg::as_policy_str -> parse_device_policy mapping (gpu_required
and gpu_preferred resolve to GpuRequired; cpu_only is rejected), closing
the previously untested conversion path.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* docs(serve): stop advertising hidden cpu_only device value

Hiding `cpu_only` from `--device` help/completion left three surfaces still
presenting it as a real choice. Align them with what the CLI now advertises:

- README serve synopsis dropped `cpu_only` from the --device value list.
- Removed the '--gpu is ignored with --device cpu_only (the model runs on CPU)'
  sentence: cpu_only is rejected (exit 1), never runs on CPU, so the line
  advertised a fallback path that does not exist (AGENTS.md §6).
- Corrected the drifted test comment to note cpu_only is #[value(hide = true)]
  and that serve_device_help_lists_match_device_policy_names compares against
  the full DevicePolicy set (hidden entries included).

Doc/comment only; no behavior change.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

---------

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* fix(dash): launcher shows live serving instances; amd-smi detection off critical path (eai-8190) (ROCm#295)

* fix(dash): seed launcher front door with live serving instances

The launcher front door built an empty AppState, so it always rendered
the idle variant even when a model was actively serving. Read the
managed-service registry (the same authority `rocm services` reads)
once per hub-loop pass and seed the AppState's instances from it.

Also treat `Ready` as serving everywhere the dashboard counts running
instances (`is_serving()`), matching the `Running`+`Ready` treatment
already used elsewhere (e.g. home.rs) -- a served model reports
`Ready`, not `Running`, so the count previously undercounted actual
serving models.

Adds apps/rocm's direct dependency on rocm-dash-core (previously only a
transitive dep) so the launcher can build `Instance`s from the
registry's `DiscoveredService` records.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* fix(dash-daemon): run amd-smi detection off the run loop's critical path

Detecting amd-smi (`amd-smi version` plus the first `system_info()`)
can take up to ~15s on real hardware. Running it inline before the run
loop's first tick blocked managed-service discovery and the first
snapshot broadcast behind it, so an already-running model did not
surface in the dashboard until GPU detection finished -- a visible
~15-20s "0 models running" lag while `rocm services` already reported
it live.

Spawn detection in the background and adopt the result via a oneshot
channel the moment it lands, without ever blocking the loop while it
is in flight. The loop now starts ticking immediately, so serving
instances appear within one discovery tick; GPU metrics fill in once
detection completes.

Adds a regression test asserting on ordering (the instance must
surface in a snapshot whose gpu_system_info is still None) rather than
wall-clock timing, since a pure "arrived within Ns" check would be
flaky under subscriber starvation.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

* test(dash): exercise off-critical-path detection and launcher front door

Address review feedback on the amd-smi-off-critical-path change.

- Gate the "amd-smi unavailable" warning behind gpu_init_done so a healthy
  host never flashes it during the detection window; settle as unavailable
  (and say so once) if the detection task ever ends without a result.
- Add an amd_smi_skip_kfd_preflight test seam so the daemon regression test
  drives a fake amd-smi through the real detection path instead of
  short-circuiting on a GPU-less CI host (the /dev/kfd guard stays mandatory
  in production). The test now genuinely fails if detection moves back onto
  the critical path, and asserts the surfaced snapshot carries no premature
  "amd-smi unavailable" warning.
- Add direct unit tests for launcher_serving_instances (ready record maps to
  a live Instance; unbound :0 record is dropped) and a behavioural launcher
  scenario driving bare rocm through a PTY to prove the front door shows
  "Serving <model>" rather than "Idle" for a live registry service.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

---------

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* Fix diagnose emitting invalid UNKNOWN render-group remediation (ROCm#302)

check_4_render_group built the render-group remediation command from the
/dev/kfd owner group obtained via `stat -c %G`. When the device GID has no
matching group name, stat prints the literal "UNKNOWN", and the existing
guard only rejected empty strings. That let UNKNOWN leak into the suggested
fix, producing `sudo usermod -a -G UNKNOWN,video "$USER"` in the plan,
summary, and --json output -- a command that fails because no such group
exists.

Reject "UNKNOWN" (case-insensitive) alongside empty values so the group
falls back to "render". This makes diagnose's fallback coincide with the
"render,video" that fix.rs hardcodes in the UNKNOWN/empty case; it does not
unify the two group sources, which are still computed independently and can
diverge when /dev/kfd has a real group name that is not "render".

Add a regression test covering both the "UNKNOWN" and "unknown" sentinel
casings falling back to the render group.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* docs(readme): reframe install channels around published release, document rocm examine, describe Configuration section

Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* docs(readme): remove rocm model from command reference

Not one of the commands being hardened for this release; team decision
was to drop it from the reference for now rather than document it.

Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* docs(conf): switch to rocm-ai flavor, enable repo and download buttons

Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* docs(conf): revert repository/download header buttons

use_repository_button and use_download_button rely on rocm-docs-core
resolving a repository_url from the current git branch. GitHub
Actions' pull_request checkout leaves a detached HEAD on the
synthetic merge ref, so get_branch() returns an empty URL, and
sphinx_book_theme crashes trying to unpack it, failing the -W build.

Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

* docs(conf): fix flavor fallback and pin repository buttons

- Broaden the known-warning suppression filter to also cover the
  "rocm-ai" flavor, which isn't in the published rocm-docs-core
  release yet and would otherwise fail the -W build even though
  rocm_docs.theme already falls back to "rocm" gracefully.
- Re-enable use_repository_button/use_download_button, pinning
  repository_url/repository_branch explicitly instead of letting
  rocm-docs-core infer them from the local git branch. CI's
  pull_request checkout leaves a detached HEAD on the synthetic merge
  ref, which resolves to an empty repository_url and crashes
  sphinx_book_theme's repository button; use_download_button was
  never actually affected by this, it only reads local source files.

Verified by building against the pinned rocm-docs-core==1.40.0 in a
clean venv (matching CI) -- build succeeds with 0 warnings.

Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>

---------

Signed-off-by: pmoutsia_amdeng <peter.moutsias@amd.com>
Signed-off-by: pmoutsias-amd <peter.moutsias@amd.com>
Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Signed-off-by: Michael Roy <michael.roy@amd.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Co-authored-by: Eugene Volen <150254791+volen-silo@users.noreply.github.com>
Co-authored-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Co-authored-by: Michael Roy <michael.roy@amd.com>
Co-authored-by: Roman <roman.sirokov@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Lemonade engine cannot start on Ubuntu 22.04 (glibc 2.35), but README and docs/wsl.md advertise it as supported

3 participants