Repository navigation
serve: a model-generic Q4NX container is served through the per-op forward, not refused - #1133
Conversation
…rward, not refused
engine#1129. `npu_model_kind()` knew two shapes of NPU model directory -- a private route (a build-time
addon such as dx) and the fast lane (a fixed set of shape-specific lane kernels) -- so a MODEL-GENERIC
container (model.q4nx + config.json + tokenizer.json + plan.json, exactly what our converter and packer
emit and what we publish) satisfied neither and was refused before the device was opened:
1bit serve: ~/models/npu/ZR1-1.5B-npu-q4km-q8 is not an NPU model directory (the fast lane needs npu/)
while the very same directory ran fine through `1bit forward-serve -m model.q4nx --kernels <dir>`. Since
Lemonade's onebit backend drives `serve -m <dir>`, every container of that shape was invisible to it -- the
published ZR1-1.5B q8 container included -- and an NPU scoreboard row for such a model read "N/A" because
of dispatch, not hardware.
The kernels are the whole test, exactly as they are for the GGUF branch that already ends in
run_forward_serve: if a full-ELF directory resolves (ONEBIT_NPU_KERNELS, <dir>/npu, or the build's
-DONEBIT_NPU_KERNELS_DIR), a model-generic container is handed to that same in-process route, --alias and
--npu-streams included. If none resolves, the refusal now names that shape and the three ways to give it
kernels instead of calling the directory "not an NPU model directory". Nothing that starts today changes
behaviour: a private-route or fast-lane directory is dispatched exactly as before, because the new branch is
only reached when npu_model_kind() says it is neither.
Measured, device-free (host GEMM, no /dev/accel0, no ticket), on the container the issue names:
shipped c25 payload serve -m ZR1-...-q4km-q8 --device npu -> refused "is not an NPU model directory"
same dir via forward-serve -> ready, " Paris. The capital of Germany is Berlin"
this branch serve -m ZR1-...-q4km-q8 --device npu -> "is a model-generic container: serving it
through the per-op forward (kernels .../zr1-elfs)", ready, and the same request answers
200 " Paris. The capital of Germany is Berlin" -- IDENTICAL to the forward-serve row
same dir, no kernels resolved -> refused, naming the container shape and
ONEBIT_NPU_KERNELS / <dir>/npu / -DONEBIT_NPU_KERNELS_DIR
same dir with --npu-streams 2 -> handed on to forward-serve: shared device
lock, ready "[npu-streams=2]" (the in-process route still refuses the flag)
Closes #1129.
bong-water-water-bong
left a comment
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 8c53876 — the gap is real, the fix is genuinely additive, and "identical to forward-serve" holds by construction rather than by measurement, which I checked in the code before looking for rows.
Why the identical-reply row is not really a measurement: the new branch does not reimplement the forward, it calls the same function with the same arguments —
std::vector<std::string> fargs = {"-m", (dir/"model.q4nx"), "--kernels", kernels, "-p", port, "--host", host};
if (!alias.empty()) { … }
if (o.npu_streams > 1) { fargs.push_back("--npu-streams"); … }
return run_forward_serve(int(fav.size()), fav.data());so serve -m <dir> and forward-serve -m <dir>/model.q4nx --kernels <dir> are literally the same program path from that line on. Your rows agree with that, and they should: the only way they could differ is if the argument vector differed.
The additive property is verified by the gate, and I checked the placement too: the block sits before npu_model_problem() (which is what threw), and it is entered only when npu_model_kind(o.model, opts) == NpuModel::None — so a private-route or fast-lane directory never reaches it. Kernels resolution is env → <dir>/npu → the compiled-in dir, and a model-generic container with none of them gets a refusal that names all three ways to provide them. --npu-streams > 1 is passed through to the forward, which is the route that legitimately supports N contexts in one process (#640) — consistent with #1094, which refused streams only on the unified in-process path.
Two things this path silently drops, and one of them contradicts a comment six lines below:
--laya-modelis skipped for model-generic containers. The new branchreturns before the#ifdef ONEBIT_LAYAblock, whose own comment says "with--laya-modelthe router is still asked (with npu its only candidate), so every device the engine serves goes through Laya." After this PR that sentence is false for exactly this class of container. The decision is trivial (npuis the only candidate), but the log line, the checkpoint and the--laya-*machinery are what the operator asked for, and losing them silently is the same shape as #1108's "a silently dropped field is what it must not do". Cheapest fix: move the model-generic branch after the Laya block, or run the scorer in the branch and log its answer.--npu-opt KEY=VALUEis dropped (the new branch passes no--opt, andforward-serve's parser has no such option). Either refuse it by name on this path or thread it through — a silently ignored flag is exactly what #1090's sibling fix (--npu-streamsrefused by name) exists to prevent.
Not reproduced by me: the device-free rows. They need a model-generic container plus a full-ELF kernels directory, and the box has had the GPU queue 10 deep with the token held and the NPU 2 deep all night; my one model-generic container here is the 0.5B pack import, whose ELF set I no longer have on disk. What I verified instead is the stronger structural claim — the code path is run_forward_serve, so a difference between the two rows would be a bug in the argument vector, which is nine lines long and reads correctly.
|
Row from the lane that filed #1129 (pi-onebp), on the published artifact rather than a test dir:
The text is the per-op forward's own text, not merely a served-looking answer. Control in the same ticket, Byte-identical, and the Two things worth writing into the PR/docs rather than leaving implicit:
Nothing else of mine is blocked by this: #1129's acceptance criterion is met on a real published container, from a binary built off this branch head. |
|
Post-merge row on the c26 boot, driven from the lane that filed #1129 (pi-onebp), same acceptance cells, new driver. Merged as One
All three arms returned the same 16 tokens, byte-identical: so "identical to Rate note, so nobody quotes these as throughput: 20–23 s for 16 tokens on the Q8 container is comparable to the earlier Harness, for whoever repeats this: |
|
Closing the last caveat on the row above: the same cells on a binary built from merged Posted rows usually stop at "the tree I measured is the merged tree". This one goes through the shipped binary:
All three serving arms again returned the same bytes: So the acceptance cells hold on Two notes on the harness, since I trip over it more than the engine does: the |
npu_model_kind()knew two shapes of NPU model directory — a private route (a build-time addon such asdx) and the fastlane (a fixed set of shape-specific lane kernels) — so a model-generic container (
model.q4nx+config.json+tokenizer.json+plan.json, exactly what our converter and packer emit, and what we publish) satisfied neither and wasrefused before the device was opened:
while the same directory ran fine through
1bit forward-serve -m model.q4nx --kernels <dir>. Lemonade's onebit backenddrives
serve -m <dir>, so every container of that shape was invisible to it — the published ZR1-1.5B q8 containerincluded — and an NPU scoreboard row for such a model read "N/A" because of dispatch, not hardware.
The change (one branch in
app/serve.cpp, ~30 lines)The kernels are the whole test, exactly as they are for the GGUF branch that already ends in
run_forward_serve: if afull-ELF directory resolves (
ONEBIT_NPU_KERNELS,<dir>/npu, or the build's-DONEBIT_NPU_KERNELS_DIR), amodel-generic container is handed to that same in-process route,
--aliasand--npu-streamsincluded. If none resolves,the refusal now names that shape and the three ways to give it kernels, instead of calling the directory "not an NPU model
directory".
Purely additive: the new branch is only reached when
npu_model_kind()says the directory is neither a private route northe fast lane, so everything that starts today is dispatched exactly as before.
Measured (device-free: host GEMM, no
/dev/accel0, no ticket)serve -m ZR1-…-q4km-q8 --device npuis not an NPU model directory (the fast lane needs npu/)(the bug)forward-serveready, answersParis. The capital of Germany is Berlinserve -m <dir> --device npuis a model-generic container: serving it through the per-op forward (kernels …/zr1-elfs)→ready→ 200Paris. The capital of Germany is Berlin— identical to theforward-serverowONEBIT_NPU_KERNELS/<dir>/npu/-DONEBIT_NPU_KERNELS_DIR--npu-streams 2forward-serve:[npu] shared device lock acquired (one of --npu-streams),ready … [npu-streams=2](the in-process route still refuses that flag — #1094)Acceptance is the one #1129 asked for: the container answering
/v1/completionswhile served as a directory, with textmatching the existing
forward-serverow.tools/copyright.py --checkrc 0;1bitbuilds with-DONEBIT_NPU=ON+ theprivate runtime.
Closes #1129.