Skip to content

os: ship the HSA runtime beside the engine's llama-server, and name that server for Lemonade - #580

Merged
bong-water-water-bong merged 3 commits into
mainfrom
1bit/os-hrx-runtime-on-data
Oct 26, 2026
Merged

bong-water-water-bong merged 3 commits into
mainfrom
1bit/os-hrx-runtime-on-data

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Stacked on #579 (same base; merge #579 first and this PR shows only its own commit). Files: os/mini/mkimage.sh, os/mini/init.sh.

First hardware boot of the signed candidate 5 (26 Oct, Secure Boot enabled, our MOK loaded): 14 of 16 release checks pass (fans controlled, GPU, NPU, Lemonade 401 keyless, chat 176 tok/s, bench, sealed sets, licences). The two onebit rows fail for reasons the VM gate cannot see, both fixed here:

  1. Lemonade cannot start the engine's server. scripts/lemonade-hrx defaults to the llama-server beside the script (hrx/llama/bin/llama-server); the image keeps it at /opt/1bit-llama. lemond logged Failed to execute: hrx/llama/bin/llama-server. init.sh now exports ONEBIT_HRX_LLAMA_SERVER=/opt/1bit-llama/llama-server, the script's own override, for lemond and for ssh logins via /etc/profile.
  2. The HRX backend never loads. With the server named, llama-server --list-devices printed (none): libggml-hrx is dlopened by ggml and NEEDS libhsa-runtime64.so.1, which was not beside it; the stage resolved only the program's NEEDED entries. Putting Lemonade's bundle on LD_LIBRARY_PATH made it load and then mixed that bundle's libloomc into the engine (undefined symbol: loomc_link_index_builder_allocate). mkimage.sh now resolves the NEEDED entries of every lib*.so in /data/1bit-llama, copies the HSA runtime's ROCm sysdeps flat beside it (relocate() makes every file look in $ORIGIN only), and fails the build if any library still misses one. /etc/1bit/libhsa prefers that copy, with Lemonade's bundle as the fallback for older data partitions.

Evidence on the board: with SVM left on and the right libraries on the path, llama-server --list-devices lists HRX0: AMD Radeon 8060S Graphics (gfx1151).

Not in this PR: the engine package inside the image is candidate 3's tarball (candidates 4 and 5 were package-only runs), built before the HRX_ALLOW_NO_SVM gate of #565, so on the CWSR-on kernel the HRX runtime answers "only systems with SVM are supported". Candidate 6 is a full build. Also not here: the engine's own default llama-server path when 1bit serve is run by hand (follow-up issue).

🤖 Generated with Claude Code

bong-water-water-bong and others added 2 commits October 25, 2026 23:09
…nestly

Three defects a hardware run of candidate 5 exposed, all in the image's own tooling rather than in
the release itself.

1. `/etc/1bit/onebit-models.txt` had no producer. release-check.sh read it and, when it was absent,
   failed with "the build writes it from server_models.json" - but nothing in the tree ever wrote it,
   so the `models` row could never pass on any image. mkimage.sh now derives it from the Lemonade
   catalogue it already stages at /opt/lemonade/bin/resources/server_models.json, in the format
   release-check parses: `mode=<live|installed>` then one `hrx <id>` / `npu <id>` per entry. Deriving
   it means it cannot drift from what Lemonade actually ships; a catalogue with no onebit entries
   writes a manifest with none, which the check reports as such instead of failing.
   The JSON walk accepts both a list of model dicts and a dict of sections, so a layout change
   produces a manifest rather than silently producing an empty one.

2. The `chat-hrx` and `chat-npu` skips blamed the environment for what is often a missing entry.
   Both now distinguish the two cases: no /dev/kfd (or /dev/accel/accel*) means the machine cannot run
   the entry, while a present device with no matching model id means Lemonade listed nothing to ask.
   On the hardware run this mattered: the row said "no AMD GPU in this VM" on a machine with a GPU.

3. The `secureboot` row skipped whenever mokutil was absent, though mokutil is not shipped - so a
   machine that had Secure Boot ON and had just booted our MOK-signed kernel was reported as unknown.
   It now reads the firmware's own SecureBoot EFI variable (byte 5 of SecureBoot-8be4df61-...), which
   needs no extra binary: 1 passes "Secure Boot enabled; the MOK-signed kernel booted", 0 passes on a
   firmware that reports it off, and only an unreadable variable stays a skip.

self-test passes (7/7 verdict cases). Apache-2.0 headers untouched.
…hat server for Lemonade

First hardware boot of the signed image (candidate 5, 26 Oct, Secure Boot on): 14 of 16 release
checks pass, and the two onebit rows fail for reasons the VM gate cannot see.

1. Lemonade's HRX recipe starts the engine through scripts/lemonade-hrx, whose default llama-server
   is the one beside the script (hrx/llama/bin/llama-server). The image keeps it at /opt/1bit-llama,
   so lemond logged "Failed to execute: hrx/llama/bin/llama-server" and the model never started.
   init.sh now exports ONEBIT_HRX_LLAMA_SERVER=/opt/1bit-llama/llama-server (the script's override),
   for lemond and for ssh logins through /etc/profile.
2. With the server named, llama-server listed no HRX0 at all: libggml-hrx is dlopened by ggml and
   NEEDS libhsa-runtime64.so.1, which was not beside it - the stage only resolved the program's own
   NEEDED entries. The only way to load it on the stick was Lemonade's bundle on LD_LIBRARY_PATH,
   which then mixed that bundle's libloomc into the engine (undefined symbol
   loomc_link_index_builder_allocate). mkimage.sh now resolves the NEEDED entries of every lib*.so
   in /data/1bit-llama as well, copies the HSA runtime's ROCm sysdeps flat beside it (relocate()
   makes every file look in $ORIGIN only), and refuses the build if any library still misses one.
   /etc/1bit/libhsa prefers that copy; Lemonade's bundle stays as the fallback for older data
   partitions.

Not fixed here: the engine's own default llama-server path when run by hand (follow-up issue), and
the image's engine package itself, which candidate 4 and 5 inherited from candidate 3 (package-only
runs) and which predates the HRX_ALLOW_NO_SVM gate (#565); candidate 6 is a full build.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 5b90513)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

579 - Partially compliant

Compliant requirements:

  • The manifest file is now generated by mkimage.sh from server_models.json
  • Skip reasons for chat-hrx and chat-npu are now correctly distinguished
  • Secure Boot detection now uses firmware variables directly

Non-compliant requirements:

  • The chat-hrx "did not answer" failure is not addressed in this PR (as noted in the ticket description)

Requires further human verification:

  • Verification that the firmware variable parsing works correctly on actual hardware
  • Confirmation that the chat-hrx "did not answer" issue is resolved in a separate change

565 - Partially compliant

Compliant requirements:

  • HRX submodules are updated to the specified commits

Non-compliant requirements:

  • No specific requirements related to the bump are explicitly tested or verified in this PR

Requires further human verification:

  • Verification that the updated HRX submodules work correctly on hardware
  • Confirmation that the long-prompt correctness probe and test-backend-ops -b HRX0 pass
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing HSA runtime dependencies

The llama_to_data function now copies HSA runtime dependencies and ROCm sysdeps, but it does not validate that all required libraries are present after copying. If a library is missing from the source paths, it could lead to runtime failures. The check at the end of the function ensures that the llama-server and all lib*.so files are fully resolved, but it does not ensure that the ROCm sysdeps are correctly copied.

for f in "$out/data.dir/1bit-llama/llama-server" "$out/data.dir/1bit-llama"/lib*.so; do
    if ldd "$f" 2>/dev/null | grep -q "not found"; then
        echo "$(basename "$f") on the data partition still misses libraries:"
        ldd "$f" | grep "not found"
        exit 1
    fi
done
Incorrect HSA runtime path resolution

The logic for resolving the HSA runtime path in init.sh prioritizes /data/1bit-llama over /data/lemonade/cache/bin/llamacpp-hrx/*/lib, which may lead to incorrect behavior if the HSA runtime is not present in the expected location. This could cause the system to fail to find the correct HSA runtime, especially in older data partitions.

hsa=$(ls /data/1bit-llama/libhsa-runtime64.so* 2>/dev/null | head -1)
[ -n "$hsa" ] || hsa=$(ls /data/lemonade/cache/bin/llamacpp-hrx/*/lib/libhsa-runtime64.so* 2>/dev/null | head -1)
[ -n "$hsa" ] && echo "$hsa" > /etc/1bit/libhsa

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 5b90513

@bong-water-water-bong
bong-water-water-bong merged commit aed8691 into main Oct 26, 2026
8 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/os-hrx-runtime-on-data branch October 26, 2026 02:37
bong-water-water-bong added a commit that referenced this pull request Oct 26, 2026
… let an unmatched glob end the image stage (#583)

Candidate 6's image stage died after the library copies with no message (exit 2). Two causes, both in
#580's block:

1. libggml-hrx does not list the HSA runtime as NEEDED - it dlopens it through
   IREE_HAL_AMDGPU_LIBHSA_PATH - so the NEEDED sweep never copied libhsa-runtime64.so.1 into
   /data/1bit-llama.
2. hsa_pkg=$(ls .../1bit-llama/libhsa-runtime64.so* | head -1) then matched nothing, and under
   set -e -o pipefail that assignment exits the script with ls's status 2, silently.

Now the runtime is staged on purpose: ONEBIT_HRX_LIBHSA from the build's CMake cache when it is set and
the file exists, else Lemonade's llamacpp-hrx copy on the stick, else a named error. Only what the
runtime resolves comes with it (its ROCm sysdeps and librocprofiler-register, about 3 MB, not the 33-file
sysdeps directory), flat beside it. The two lookups are globs with existence tests. The final check runs
ldd over llama-server and every lib*.so* on the data partition and fails with the missing soname.

Proved in isolation on candidate 6's work/build + work/os (strixhalo): 52 files, 82 MB, zero 'not found'
across every staged file; libhsa staged from the build's own runtime with 9 libraries it resolves.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
bong-water-water-bong pushed a commit that referenced this pull request Oct 26, 2026
…gs, install, shell)

First hardware boot of candidate 5 left the console unusable: kernel lines over a bare "# " prompt
and nothing to do with it. This is what the owner asked for instead - a dialog-style full-screen
menu - as one POSIX sh script, os/mini/1bit-menu, with no new binary in the image (there is no
dialog, whiptail or ncurses on the stick, and terminfo is not guaranteed on a text console, so it
draws with ANSI escapes in the 16-colour set).

tty1 only. init.sh starts the menu in place of the bare `sh -i` loop; ttyS0 (the serial line the
QEMU gate drives) and ssh keep exactly the plain shell they have today, so no prompt matcher on the
gate changes, and the gate builds nothing new: the same init.sh reaches the same serial shell.

The menu shows the banner facts it was previously printed as text (edition, LIVE/INSTALLED, address,
Lemonade URL, where the API key is) and drives only commands the appliance already has - it starts
neither the engine nor Lemonade, which are up by the time init gets here:

  Chat     pick a model from the Local models `lemonade list` reports (the onebit ones marked) and
           run `lemonade chat <id>` in the terminal; back on exit
  Models   `lemonade list` and the GGUFs on /data/models, paged by a built-in pager
  Status   the tail of /data/selftest.log and the PASS/FAIL/SKIP rows of the LATEST
           /data/release-check.log run (each run is delimited), refreshing once a second; r runs
           `1bit-release-check` again and shows its output
  Logs     tail -40 of /tmp/lemond.log, /data/boot.log and /tmp/fans.log, any key refreshes
  Install  the internal NVMe with its size, model and partition table, plus 1bit-install's own
           refusal text (run without --yes, so the warning cannot drift from the tool); typing
           INSTALL is required, then `1bit-install /dev/nvme0n1 --yes`, then an offer to reboot.
           When the machine is installed it shows what it runs from instead
  Shell    `sh -i` with PS1 "1bit-os:\w # "; exit comes back to the menu
  Power    reboot or power off, confirmed with y

Keys: up/down or j/k, 1-7 jump, Enter chooses, q quits to the plain shell. `1bit-menu --shell`, or
holding q as it starts, skips the menu entirely.

The console can never be swallowed. The menu restores the terminal on EXIT, INT, TERM and HUP, and
init runs `stty sane` on tty1 before every restart (an addition asked for after review: a crash must
not leave raw mode behind). Any exit other than 0 restarts the menu; q means "give me the shell" and
drops to it. Two details found while proving this:

  * init has to run that loop in the foreground, not as `prog &`: POSIX has an asynchronous list
    ignore SIGINT and SIGQUIT, and the child inherits it - so a menu started with & could not be
    interrupted, and neither could the lemonade chat or the shell it runs. init has nothing else to
    do once the appliance is up, and both loops never return.
  * key reads wait in 0.5 s slices instead of blocking, so a menu killed with SIGKILL cannot leave a
    dd blocked in a read; a straggler holds tty1's session and then nothing can take the tty again.

Rebased onto baf5996 (#583): main's #582 (ignore_loglevel/printk off before the console, and
PS1="1bit-os:\w # " on tty1, ttyS0 and ssh through /etc/profile), #580 (ONEBIT_HRX_LLAMA_SERVER,
the libhsa preference), #583 (explicit HSA staging) and #579 (the onebit manifest) are all kept. The
menu replaces only the tty1 shell loop; the plain-shell fallback below it keeps #582's PS1, and the
menu's Shell entry sets the same one. The printk lines now pay off twice: a boot line painted over the
frame is a mess the menu can only clean up on the next keypress.

Proof, on ryzen, no GPU and no strixhalo reboot: bash -n / sh -n / dash -n clean; run in an 80x25
tmux pane with TERM=linux and a stub `lemonade` (plus stub 1bit-install and 1bit-release-check) on
PATH - it draws, navigates, and runs Chat, Models, Status (including re-running the check), Logs,
Install and Shell, and comes back from each; q and Ctrl-C both leave the terminal with icanon and
echo back. The init loop itself was driven on a pty nothing else owns: a SIGKILLed menu leaves raw
mode behind, the loop's `stty sane` restores it and the menu comes back, Ctrl-C is restarted by the
loop, and q falls through to the plain shell.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bong-water-water-bong added a commit that referenced this pull request Oct 26, 2026
…gs, install, shell) (#584)

First hardware boot of candidate 5 left the console unusable: kernel lines over a bare "# " prompt
and nothing to do with it. This is what the owner asked for instead - a dialog-style full-screen
menu - as one POSIX sh script, os/mini/1bit-menu, with no new binary in the image (there is no
dialog, whiptail or ncurses on the stick, and terminfo is not guaranteed on a text console, so it
draws with ANSI escapes in the 16-colour set).

tty1 only. init.sh starts the menu in place of the bare `sh -i` loop; ttyS0 (the serial line the
QEMU gate drives) and ssh keep exactly the plain shell they have today, so no prompt matcher on the
gate changes, and the gate builds nothing new: the same init.sh reaches the same serial shell.

The menu shows the banner facts it was previously printed as text (edition, LIVE/INSTALLED, address,
Lemonade URL, where the API key is) and drives only commands the appliance already has - it starts
neither the engine nor Lemonade, which are up by the time init gets here:

  Chat     pick a model from the Local models `lemonade list` reports (the onebit ones marked) and
           run `lemonade chat <id>` in the terminal; back on exit
  Models   `lemonade list` and the GGUFs on /data/models, paged by a built-in pager
  Status   the tail of /data/selftest.log and the PASS/FAIL/SKIP rows of the LATEST
           /data/release-check.log run (each run is delimited), refreshing once a second; r runs
           `1bit-release-check` again and shows its output
  Logs     tail -40 of /tmp/lemond.log, /data/boot.log and /tmp/fans.log, any key refreshes
  Install  the internal NVMe with its size, model and partition table, plus 1bit-install's own
           refusal text (run without --yes, so the warning cannot drift from the tool); typing
           INSTALL is required, then `1bit-install /dev/nvme0n1 --yes`, then an offer to reboot.
           When the machine is installed it shows what it runs from instead
  Shell    `sh -i` with PS1 "1bit-os:\w # "; exit comes back to the menu
  Power    reboot or power off, confirmed with y

Keys: up/down or j/k, 1-7 jump, Enter chooses, q quits to the plain shell. `1bit-menu --shell`, or
holding q as it starts, skips the menu entirely.

The console can never be swallowed. The menu restores the terminal on EXIT, INT, TERM and HUP, and
init runs `stty sane` on tty1 before every restart (an addition asked for after review: a crash must
not leave raw mode behind). Any exit other than 0 restarts the menu; q means "give me the shell" and
drops to it. Two details found while proving this:

  * init has to run that loop in the foreground, not as `prog &`: POSIX has an asynchronous list
    ignore SIGINT and SIGQUIT, and the child inherits it - so a menu started with & could not be
    interrupted, and neither could the lemonade chat or the shell it runs. init has nothing else to
    do once the appliance is up, and both loops never return.
  * key reads wait in 0.5 s slices instead of blocking, so a menu killed with SIGKILL cannot leave a
    dd blocked in a read; a straggler holds tty1's session and then nothing can take the tty again.

Rebased onto baf5996 (#583): main's #582 (ignore_loglevel/printk off before the console, and
PS1="1bit-os:\w # " on tty1, ttyS0 and ssh through /etc/profile), #580 (ONEBIT_HRX_LLAMA_SERVER,
the libhsa preference), #583 (explicit HSA staging) and #579 (the onebit manifest) are all kept. The
menu replaces only the tty1 shell loop; the plain-shell fallback below it keeps #582's PS1, and the
menu's Shell entry sets the same one. The printk lines now pay off twice: a boot line painted over the
frame is a mess the menu can only clean up on the next keypress.

Proof, on ryzen, no GPU and no strixhalo reboot: bash -n / sh -n / dash -n clean; run in an 80x25
tmux pane with TERM=linux and a stub `lemonade` (plus stub 1bit-install and 1bit-release-check) on
PATH - it draws, navigates, and runs Chat, Models, Status (including re-running the check), Logs,
Install and Shell, and comes back from each; q and Ctrl-C both leave the terminal with icanon and
echo back. The init loop itself was driven on a pty nothing else owns: a SIGKILLed menu leaves raw
mode behind, the loop's `stty sane` restores it and the menu comes back, Ctrl-C is restarted by the
loop, and q falls through to the plain shell.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
bong-water-water-bong added a commit that referenced this pull request Oct 26, 2026
…uth (#581) (#592)

Candidate 8 on the board (26 Oct, Secure Boot on): every row passes except chat-hrx, with the same line as
candidate 5 - lemond spawns `1bit serve` itself and the engine's default server is the build's relative
path, which does not exist in the image: "Failed to execute: hrx/llama/bin/llama-server". #580's env
export did not help because serve reads ONEBIT_LLAMA_SERVER while the launcher script reads
ONEBIT_HRX_LLAMA_SERVER.

app/serve.cpp: default_llama_server() resolves, first hit wins: ONEBIT_LLAMA_SERVER,
ONEBIT_HRX_LLAMA_SERVER, the build's own server beside the executable when it exists,
/opt/1bit-llama/llama-server (where the image keeps it), then llama-server on PATH.
os/mini/init.sh: exports both names.

os/mini/board.sh: the boot said "EC profile only (no SMU cap: ryzenadj absent or /dev/mem not writable)"
on a boot where the cap HAD been written. Measured on the stick: `ryzenadj --stapm-limit=90000
--slow-limit=90000 --fast-limit=105000` exits 0 ("Successfully set ..."); only the pm-table read that
`--info` needs is refused by STRICT_DEVMEM, and power_status keyed its verdict on `--info`. The apply path
now records the written cap in /run/1bit-smu-cap and power_status reports that; the fallback wording
names the real case (present but not applied / no tool).

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant