Repository navigation
1bit serve: one model on any device behind one OpenAI-compatible API - #21
Merged
Merged
Conversation
The engine as a backend a host launches (geramyL's point: embed the engine into Lemonade, not Lemonade into the engine). One model per process: /health (+/v1/health) 503 -> 200, /v1/models, /v1/chat/completions (SSE or not), /v1/completions. - NPU model directory -> the NPU fast lane in process (unified.cpp, which now also answers /health). - .gguf -> this build's llama-server (Vulkan0 or HRX0) or ZINC as a private loopback child; OpenAI routes forwarded, streaming relayed as it arrives, replies carry the served name, and ZINC gets requests without 'model'. - HRX: sets IREE_HAL_AMDGPU_LIBHSA_PATH to TheRock's libhsa itself (--hrx-libhsa, the build's copy, or /opt/rocm-therock). - tests/serve_e2e.sh + ctest serve_e2e_<device> (-DONEBIT_SERVE_TEST_GGUF). Verified on Strix Halo with Qwen3-0.6B Q4_K_M: vulkan, hrx and zinc all PASS (health, models, 'Paris.' under the served name, SSE streaming). The NPU route is not yet run through serve_e2e. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The e2e test left llama-server/zinc children behind: SIGTERM ended serve without its destructor, and nothing could clean up after SIGKILL. The child is now forked with PR_SET_PDEATHSIG (the kernel ends it when serve dies, SIGKILL included), and SIGTERM/SIGINT stop the server and the child cleanly. Verified on Strix Halo: after SIGTERM and after SIGKILL of serve, the child is gone; serve_e2e vulkan still passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on the NPU serve passes --alias through to the in-process NPU server (unified now takes --alias), and the NPU route honours chat_template_kwargs.enable_thinking=false the way Qwen3's template does (an empty think block after the assistant prefix), so every device answers alike. serve_e2e npu PASS on Strix Halo (Qwen3-0.6B NPU model dir: 'Paris' under the served name, 33 SSE chunks). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
enabled auto-merge (squash)
September 23, 2026 22:34
bong-water-water-bong
added a commit
that referenced
this pull request
Sep 26, 2026
…pp-vulkan 546800c) (#161) Fork PR #21: ONEBIT_MOE_SUBST=r lets a resident expert among the router's next choices stand in for a missing one scoring at most 1/r times higher, in the same host sync as the remap. On Qwen3-Coder-30B, r 0.5 cut drive reads 20-27% for a KL divergence of 0.008-0.012, and decode gained 10-35% at 1536-3072 slots. Off by default. moe/ExpertCache gains resident(), as in the fork's copy. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 21, 2026
bong-water-water-bong
added a commit
that referenced
this pull request
Oct 23, 2026
…tor sync) (#520) third_party/lemonade: a98f3e23d393 -> 27abcc24da99. Fork PR #21 merged upstream main 49e63790 (the HTTP server refactor, #3771: route classes, core components, generated API docs) and #20 (three Qwen3 NPU catalog entries). Onebit hooks checked on the new route classes; hardware verify on strixhalo (90 W cap, PCM pad): 27B MTP one-hop load 8-11 s, chat 30.9-31.2 tok/s, lemonade bench chat-short TTFT 827-835 ms / TPS 28.0-28.4, Qwen3.5-4B 64.6 tok/s, NPU entry ready in 977 ms, Guest tier 0 vs 8 onebit entries without/with the engine binary. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
bong-water-water-bong
pushed a commit
that referenced
this pull request
Oct 23, 2026
…he new server reads it
On the Lemonade this image now carries (fork main after the #3771 server refactor), a top-level
"onebit_backend" in user_models.json is not read. Measured with a wrapper around the engine binary:
lemond passed `serve -m <dir> --port 8001 --device auto --alias ... --ctx-size 32768` for a register
that carried "onebit_backend": "npu", so the directory only reached the NPU because the engine's own
routing recognised a .q4nx set. The catalog's NPU entries carry the option as
recipe_options.onebit_backend, and that is what the server reads, so register the stick's directories
the same way.
Also notes in the comment that Lemonade's own catalog carries the three NPU entries by name
(Qwen3-{0.6B,1.7B,4B}-NPU-1bit): they list from the image, and this registration makes the copy on the
stick loadable without the Hub.
Verified on the merged server separately (fork PR #21): with the option honoured the load reports
`1bit unified: loading ... on the NPU`, ready in 977 ms, and the chat answers `NPU OK.` at 52.8-69.0 tok/s
(90 W cap, PCM pad (23 Oct)). Image-level VMTEST follows once engine #520 (the pin bump) merges.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bong-water-water-bong
added a commit
that referenced
this pull request
Oct 23, 2026
…y so the way the server reads it (#523) * os image: an NPU model directory says so in recipe_options, the way the new server reads it On the Lemonade this image now carries (fork main after the #3771 server refactor), a top-level "onebit_backend" in user_models.json is not read. Measured with a wrapper around the engine binary: lemond passed `serve -m <dir> --port 8001 --device auto --alias ... --ctx-size 32768` for a register that carried "onebit_backend": "npu", so the directory only reached the NPU because the engine's own routing recognised a .q4nx set. The catalog's NPU entries carry the option as recipe_options.onebit_backend, and that is what the server reads, so register the stick's directories the same way. Also notes in the comment that Lemonade's own catalog carries the three NPU entries by name (Qwen3-{0.6B,1.7B,4B}-NPU-1bit): they list from the image, and this registration makes the copy on the stick loadable without the Hub. Verified on the merged server separately (fork PR #21): with the option honoured the load reports `1bit unified: loading ... on the NPU`, ready in 977 ms, and the chat answers `NPU OK.` at 52.8-69.0 tok/s (90 W cap, PCM pad (23 Oct)). Image-level VMTEST follows once engine #520 (the pin bump) merges. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * os vmtest: prove what the image is for - live banner, self-test, the NPU names, the guest tier The VM test checked that the appliance boots and that Lemonade serves; it did not check the four things this image exists to do, so a regression in any of them could ship green. Added, in the order the owner meets them: * the LIVE banner reaches the console (the serial line is the VM's screen - a blank screen is what made the real appliance look dead); * `1bit-selftest` runs from the image and reports the LIVE mode; the VM has no Radeon GPU and no XDNA NPU, so the NPU line is reported as it comes out rather than required to pass - the NPU PASS belongs to the real boot; * Lemonade lists the three catalog NPU entries by name (Qwen3-{0.6B,1.7B,4B}-NPU-1bit). Listing only: the NPU cannot run under QEMU, and the check says so; * the Guest tier still behaves as upstream v2026.43.3 defines it: with the engine binary moved aside, a second lemond reports 0 onebit entries, and the binary is put back afterwards. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * os vmtest: check the live banner and the NPU catalog where they can actually be seen Two of the assertions I added were wrong, and the first VM run said so: * "the live banner reaches the console" matched grub's own menu line ("1bit OS v2026.43 ..."), not init's banner - grub says "1bit OS" too. It now matches init's live-mode line ("LIVE. You are root, on RAM"), which say() writes to /dev/console (this serial line in the VM) and to tty1, and it falls back to the /etc/1bit/live marker rather than passing on the wrong line. * "Lemonade lists the three NPU entries" cannot pass under QEMU: this Lemonade hides every onebit model when the machine has no AMD GPU and no NPU - the stick's own onebit register answered model_not_found in the VM, not just the catalog entries. The check now reads the IMAGE's own catalog (/opt/lemonade/bin/resources/server_models.json) for the three names and for recipe_options.onebit_backend: npu, which is what "carries them by name" actually means, and prints a line saying why the listing cannot show them here. The rest of the run was good: SSH, the data partition, the HRX JIT cache, Lemonade's backends on the stick (11 of them, no download), the API key refusing an unauthenticated request, a CPU chat answering "Paris.", and the Guest tier reporting 0 onebit entries with the engine binary moved aside. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * os vmtest: match init's own live line, and read the catalog option the way the file spells it Two leftovers from the previous commit, both caught by running it: * the banner check looked for a string that only ever goes to tty1, so it could not see it over the serial console. init's menu banner - "1bit OS v2026.43 - LIVE: runs from RAM, nothing is installed, disks are never touched", written through mc() to /dev/console and tty1 - is the line that actually appears on the VM's screen. * the recipe_options check used a case pattern whose quoting did not survive into the script, so it failed against a catalog that does carry "recipe_options": {"onebit_backend": "npu"}. It greps the catalog text now. VMTEST PASS on the pinned Lemonade (27abcc24), 90 W cap, PCM pad (23 Oct): SSH up 27 s after a UEFI boot from the USB image; web app HTTP 200 text/html (the prefix now builds it with the nvm node); KaTeX font 200 woff2; 401 without the key; the stick's GGUF lists and answers "Paris." on llamacpp:vulkan; 0 downloads (the backends came from the stick); the live banner is on the console; 1bit-selftest runs and reports the LIVE mode, with the NPU line reported as FAIL because QEMU has no NPU (the NPU PASS is the real boot's); the image's catalog carries Qwen3-{0.6B,1.7B,4B}-NPU-1bit with recipe_options.onebit_backend: npu; the Guest tier reports 0 onebit entries with the engine hidden. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Oct 23, 2026
bong-water-water-bong
added a commit
that referenced
this pull request
Oct 30, 2026
…ag (#642) The daily fork sync builds its fallback branch from the release tag instead of from the fork's main: git checkout -q -b "sync/${tag}" "$up" Since fork #21 the fork's main follows upstream's *main* (the #3771 server refactor), while a release tag is cut from the older release line. A branch at the tag is therefore not "main plus a release": it is the release's whole tree, and the PR built from it asks a person to resolve a revert. Fork #27 arrived that way - 524 files, +56318/-128209, server.cpp 8456 lines against main's 327 - which reverts 44 upstream commits and moves our onebit registration back into the release's monolithic server.cpp. The onebit guard never gets a say: it only runs on the clean-merge path, and on that tree src/cpp/server/backends/onebit/ does not exist at all. The same is true of the `merge-base --is-ancestor "$up" HEAD` guard: with a main-based fork it is false for every release tag, so the workflow would produce that revert branch every Friday. Port the release's own contribution instead: cherry-pick the commits `git cherry` marks '+' (the ones main lacks and has no patch-equivalent of) onto main, one squashed commit for the port. A commit the fork also changed keeps the fork's side and is listed in the PR, so the branch can never revert main; a person finishes the port from the release's version, which stays in the picked commit. The tag is never merged and never checked out as a branch tip. Idempotency: a ported release is neither an ancestor nor patch-equivalent (it lands in the refactored files, e.g. services/model_options.cpp rather than the release's server.cpp), and GitHub deletes the sync branch on merge, so the merged PR is what the guard looks at - `gh pr list --state merged --head sync/<tag>`. An open sync branch short-circuits the run too. Rehearsed against the real refs, not just read: - Port on last week's main (9f66d8417 + v2026.44.1): branch = main + the release's 7 non-colliding files (+41/-15), no deletions; conflicts reported as docs/api/lemonade.md and src/cpp/server/server.cpp - the same two the old workflow named in fork #27 - and both kept on the fork's side. - Guard on this week's state: `gh pr list --state merged --head sync/v2026.44.1` returns 27, so the next run is a no-op instead of a second port. - YAML parses, and the step's shell passes `bash -n`. Not fixed here: the release line still differs in layout until upstream's release branch catches up, so a person still resolves the listed files (for 44.1 that was services/model_options.cpp and docs/api/openai.md). The workflow cannot verify a port builds - this CI has no GPU - so the pin's own check stays Lemonade's LLM suite through the onebit recipe on Strix Halo (docs/releases.md, "Bumps"). Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First step of the inversion geramyL proposed: the engine is a backend Lemonade launches, not a host that embeds Lemonade. It exposes only an OpenAI-compatible API.
It serves one model per process, which is the contract Lemonade launches backends with:
/healthand/v1/health: 503 while loading, then 200/v1/models/v1/chat/completions, streamed (SSE) or not/v1/completions.ggufwith--device vulkanorhrx.ggufwith--device zincmodel.servepointsIREE_HAL_AMDGPU_LIBHSA_PATHat TheRock's libhsa itself.auto= Vulkan for GGUF until Laya routes per request.Verified on Strix Halo (Qwen3-0.6B Q4_K_M,
tests/serve_e2e.sh)All four devices pass, the NPU included. The NPU route now also takes
--aliasand honourschat_template_kwargs.enable_thinking=false, like the GPU backends. After a SIGTERM or SIGKILL ofserve, no backend is left running (the child gets PR_SET_PDEATHSIG).Next, in separate PRs: remove the embedded Lemonade (
third_party/lemonadeand its deltas) from the engine, and open the upstream Lemonade PR, a recipe that launches1bit serve.🤖 Generated with Claude Code