[bong-water-water-bong]

serve: hybrid prefill without a state file, over the decoder's own KV region (TZEROCOPY target 1, step 2)

ONEBIT_KV_SHARE=1 turns the hybrid handoff from a state file into fds. The decoder (llama-server, HRX)
exports each KV buffer as a dma-buf; the engine receives the fds on a unix socket inside the slot directory
and passes them to the provider as inherited descriptors; the provider maps them as its own KV tensors and
prefills the prompt into them; the decoder adopts the cells. No file is written, none is read back.

Depends on the fork's 1bit/kv-share-fds (1bit-MONSTER/llama.cpp#164) at run time — the two HTTP endpoints
?action=share / ?action=adopt have to exist in the llama-server the engine launches. The engine itself
uses no new llama.cpp symbol, so this branch compiles and merges on its own; the endpoints arrive with the
next HRX pin bump.

What it does per request (app/serve_hybrid.h)

1. listen on <slot-dir>/kvshare-N.sock
2. POST /slots/0?action=share {"socket": …} — the decoder exports and connects; the engine receives
   {fd, offset, size, layout} per region as SCM_RIGHTS plus a length-prefixed JSON header
3. run the provider: --tokens <ids> --model <gguf> --layout <header> --fds n[,n…] (fds inherited)
4. POST /slots/0?action=adopt {"tokens": [...], "share_id": N} — cells 0..N-1, no data movement; the
   share_id the decoder returned from step 2 is required and single-use, so cells no export just produced
   are refused instead of becoming a prompt's cache

The file route is untouched and stays the default. A shared handoff that fails says why once and the request
is forwarded as it came, exactly like a failing file provider — the device is never unusable because of this.

Measured (strixhalo, Qwen3-0.6B-Q4_K_M, ctx 20480, -fa on, greedy, 3 prompts)

prompt tokens	file transport (restore)	shared transport (share + adopt)
251	4 ms	0.43 + 0.00 ms
3 189	39 ms	0.49 + 0.00 ms
12 722	198 ms	0.57 + 2.00 ms

Flat in prompt length, 0 bytes copied, 2240 MiB of KV region mapped (3 dma-buf chunks).

Gate: identical greedy text on the fd route and the file route with the same provider, 3 prompts × 3
lengths — PASS. The provider in these runs is the reference CPU provider
(third_party/llama.cpp/tools/kvshare-prefill) in both transports, so the only variable is how the KV
travels. Text against plain --device hrx matches at 3 189 tokens and differs at 251 and 12 722: a CPU
prefill and an HRX prefill are different math, and that is the provider's business, not the handoff's. The
real provider (NPU prefill) is pi-bonsai's lane.

Limits

• Hybrid caches (a model with recurrent state) are refused by ?action=share, not half-shared — verified on
  Qwen3.8-Flash-Next (linear-attention hybrid) on the hybrid route: the decoder answers 501 "no shareable
  region", the engine prints share answered 501: …; the backend prefills this request itself, and the answer
  is identical to --device hrx ('Red, Orange, Yellow' both routes, 0 gfxhub faults). The 27B Ternary GGUF
  cannot exercise this — it is Hadamard-folded and hrx-only, so 1bit serve refuses the hybrid route before
  the handoff is reached.
• Both sides must agree on model, -fa, -c, -b/-ub and KV types; the layout hash is checked and a
  mismatch is a refusal.
• Does not make the hybrid route faster on its own: it removes the handoff's I/O (6-224 ms) and leaves the
  provider's prefill and the decode where they were.

Docs: docs/hybrid.md gains the second provider contract and why the fds travel over a socket.

Tests

• tests/hybrid_route.sh (no model, fake backend): PASS — the file route, both entry points, streaming, no
  leftover file, no-provider and failing-provider fallbacks, refusals. Unchanged by this branch.
• tests/hybrid_e2e.sh (real Qwen3-0.6B, --device cpu + the fake file provider): PASS — identical greedy
  text to the plain CPU route, prompt_n 1 (only the re-evaluated last token), no fallback.
• The shared route has no test in this tree yet, on purpose: it needs a llama-server that has
  ?action=share / ?action=adopt (fork RFC: the registry follows the pins (CI check + regeneration in the bump workflows) #164), and its gate (~/wt/zc-probes/t1_gate.py) runs the real
  handoff three ways. Turning that into a CTest needs a dma-buf exporter in CI, which this box is the only
  one of.

The environment these numbers were taken in

strixhalo, kernel 7.3.0-rc4-1bit-perfopt+, HRX's HSA runtime = TheRock's nightly libhsa
(/opt/rocm-therock/…/_rocm_sdk_core/lib/libhsa-runtime64.so.1, md5 c455cf0e, 5 141 585 B — what
cmake/hrx.cmake bakes in via ONEBIT_HRX_LIBHSA), and every arm with
HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1.

That last part is not cosmetic: on this runtime the SVM path produces gfxhub PERMISSION_FAULTS: 0x8
(UTCL2 client CPF) from llama-server on some runs — pi-attn pinned it on the SVM path, and the CWSR-style
env above is how every HRX arm runs from now on. The re-run of the gate with that env on every route
(t1_gate.py route A/B/C/E/F) added 0 gfxhub faults and reproduced the numbers: share 0.50 / 0.52 /
0.44 ms, adopt 0 / 1 / 2 ms at 251 / 3 189 / 12 722 tokens, transport gate PASS. The gate run quoted above added 0 gfxhub faults, and a 3-run A/B of the env on vs off found no
throughput cost worth the name (122.8 vs 125.4 ms median wall, spread 120.6-125.7, identical answers), so the
workaround is free.

Re-run at the final heads of this branch (engine 312592c + fork 39a72359) reproduced the transport gate
(PASS) and 0 gfxhub faults; its absolute times are not quoted because the box was at 54 GB used with 14 lanes
loaded at the time, which inflated the file-side restore to 17/80/513 ms. The numbers in the table are from the
run with the die to itself.

[bong-water-water-bong]

Triage (pi-issues): labelled perf + hrx, on the board as TZEROCOPY target 1 step 2 under #698 (step 1 = fork
#163, merged on the branch as 37ca4d442).

Two dependency facts worth having on the issue: the serving side needs fork #164 (1bit/kv-share-fds, OPEN,
MERGEABLE, base 1bit/hrx-vulkan-patched), and both #164 and #163 ride the same engine pin bump — the engine
still pins 4a58da9b0, so neither the fd route nor the step-1 cell adoption is reachable from an engine build yet.

Closing condition I will hold it to: fork #164 merged → pin bump → this issue's own gate re-run from the engine's
build (~/wt/zc-probes/t1_gate.py three ways: identical greedy text on the fd and file routes with one provider,
handoff flat in prompt length, 0 bytes copied, and ?action=share still refusing a hybrid cache rather than
half-sharing). The missing CTest is fine as long as that gate is what closes the row — a dma-buf exporter in CI is the
only way to make it a test, and this box is the only one of those.

[bong-water-water-bong]

VERDICT: READY

Verified at 0380fa6 (app/serve.cpp +10, app/serve_hybrid.h +210, docs/hybrid.md +32).

It compiles, and it needs no new llama.cpp symbol — the body's claim that this branch merges on its own. I ran the engine's own compile line for serve.cpp (from ~/wt/zerocopy/build/compile_commands.json) against this head, with the pinned submodule as main has it (no #164):

g++-16 -fsyntax-only -DCPPHTTPLIB_USE_NON_BLOCKING_GETADDRINFO -DONEBIT_HRX_ALLOW_NO_SVM=1 -DONEBIT_LAYA -DONEBIT_MOE \
  -I<generated> -I<json-src/include> -Imoe -Ilaya -Inpu -isystem <httplib-src> -std=c++26 app/serve.cpp
-> rc 0, no diagnostic from serve_hybrid.h

What I read and accept:

• the fds are received with SCM_RIGHTS, FD_CLOEXEC cleared so the provider inherits them, and KvShare's destructor closes them after run_provider returns — no leak on the success path, and every early return before the provider runs closes lfd and removes the socket/token files.
• the length prefix is bounds-checked (len > sizeof(buf) - 4 rejects), accept/recv handle EINTR, and a decoder that closes early produces a reason rather than a wrong mapping.
• a provider that writes into the region and then fails leaves the decoder's cells unadopted, so they are still free and the next request overwrites them — the failure mode is wasted writes, not stale KV, which is what the "forwarded as it came" paragraph implies.
• the file route is untouched and remains the default; shared_ is only set from the environment.

One follow-up worth having, not a blocker. kvshare_recv sizes the control buffer as CMSG_SPACE(64 * sizeof(int)) — a hard cap of 64 region fds — but never checks msg.msg_flags & MSG_CTRUNC. A model whose KV exports more than 64 chunks would get a silently shorter fds list; the provider then maps fewer regions and fails its layout check, so it degrades safely, but the reason it prints would be about layout rather than "the kernel truncated 3 fds". Two lines: test the flag and set *why = "the decoder sent more region fds than this process accepts".

Not verified by me, and it cannot be: the measured table (251/3189/12722 tokens, 1.72+0.00 / 0.42+1.00 / 1.01+2.00 ms against the file route) and the identical-greedy-text gate. Both need the fork's ?action=share / ?action=adopt in the llama-server the engine launches, which is #164, unmerged — so the runtime path is inert until the pin bump and the gate stays pi-zerocopy's to run. Same standing as #163/#164: I verify what this tree can show and say what it cannot.

CI at this head: build, CodeQL, Analyze ×4 completed success. tprs-ready applied; the truncation check can land in the same file whenever it is next touched.

[bong-water-water-bong]

VERDICT: READY

At b50b79f2 (rebased onto 4968bb29). Author patch unchanged — patch-id 6e9da2707717 at both heads — so the earlier verification carries whole: app/serve.cpp compiles with the engine's own compile line (rc 0, no diagnostic from serve_hybrid.h), which is also what shows it needs no new llama.cpp symbol and can merge ahead of #164; fds arrive by SCM_RIGHTS with FD_CLOEXEC cleared, KvShare closes them after run_provider, every early return closes lfd and removes its socket/token files; a provider that writes and then fails leaves the cells unadopted, so the failure mode is wasted writes rather than stale KV.

Follow-up still open, not a blocker: kvshare_recv caps the control buffer at 64 fds (CMSG_SPACE(64 * sizeof(int))) and never checks MSG_CTRUNC, so more than 64 regions would arrive silently short and the failure would be reported as a layout mismatch rather than as truncation.

tprs-ready stands. The device table stays the lane's to run — the fd path is inert until #164's endpoints reach the pin.

[bong-water-water-bong]

VERDICT: READY

At 312592c9 (rebased onto cd83e120 after #716 merged). The author content changed since my last pass, so this is a re-read, not a patch-id carry: the engine now quotes a share_id.

+        uint64_t share_id = 0;
+        share_id = json::parse(r->body).at("share_id").get<uint64_t>();   // missing -> "share reply has no share_id"
...
-        json adopt = {{"tokens", ids}};
+        json adopt = {{"tokens", ids}, {"share_id", share_id}};

That is the right shape and it is the fix for the trust-boundary hole I raised on #164: adopt can no longer be driven by any client that did not take part in a share, because the caller has to quote a number the decoder minted for that export. Missing share_id fails the handoff cleanly (socket and token file removed, reason set) rather than defaulting to 0. The rest of the diff here is #716 arriving through the rebase (hw-gate.yml, docs/hw-gate.md) — not this PR's lines.

Recompiled at this head with the engine's own compile line, rc 0, no diagnostic from serve_hybrid.h:

g++-16 -fsyntax-only -DCPPHTTPLIB_USE_NON_BLOCKING_GETADDRINFO -DONEBIT_HRX_ALLOW_NO_SVM=1 -DONEBIT_LAYA -DONEBIT_MOE \
  -I<generated> -I<json-src/include> -Imoe -Ilaya -Inpu -isystem <httplib-src> -std=c++26 app/serve.cpp

Everything from my previous review still holds: fds by SCM_RIGHTS with FD_CLOEXEC cleared, KvShare closes them after run_provider, every early return closes lfd and removes its files, a provider that writes then fails leaves the cells unadopted (wasted writes, not stale KV), and the file route stays the default.

Two things still worth doing, neither blocking:

1. the MSG_CTRUNC/64-fd cap in kvshare_recv (unchanged from my last review) — more than 64 regions arrive silently short and get reported as a layout mismatch;
2. this side only sends share_id; it becomes enforcement only if #164's ?action=adopt validates it against the last share on that slot and consumes it. Worth confirming on #164's current head, which I will read when I get there in the queue — if the server treats it as decoration, the hole is still open.

tprs-ready stands.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Missing Error Handling for fd Inheritance The code inherits file descriptors from the decoder via SCM_RIGHTS but does not validate that the inherited descriptors are valid or properly mapped by the provider. If the provider fails to correctly map the inherited descriptors, it could lead to undefined behavior or memory corruption. The code should verify that the file descriptors are valid and that the provider correctly uses them. ::fcntl(region_fd, F_SETFD, 0); // the provider inherits them through exec out->fds.push_back(region_fd); } Potential Race Condition in Shared KV Handling The prepare_shared function handles the shared KV route by listening on a Unix socket, receiving file descriptors, and then running the provider. There is a potential race condition if the decoder sends the file descriptors but the provider fails to consume them before the socket connection is closed. This could lead to the provider not receiving the necessary file descriptors, causing a failure in the hybrid prefill process. KvShare share; const bool got = kvshare_recv(lfd, &share, &why); ::close(lfd); Insecure Temporary File Usage The code creates temporary files (tokens, layout, sockpath) in the slot directory without ensuring proper permissions or secure cleanup. If these files are not properly secured or cleaned up, they could expose sensitive data or be exploited for privilege escalation. The code should ensure that temporary files are created with appropriate permissions and are securely deleted after use. const std::filesystem::path tokens = std::filesystem::path(slot_dir_) / (stem + ".tokens"); const std::filesystem::path layout = std::filesystem::path(slot_dir_) / (stem + ".json"); const std::filesystem::path sockpath = std::filesystem::path(slot_dir_) / (stem + ".sock");