[bong-water-water-bong]

kv share: the region crosses processes as fds (target 1, step 2)

#163 made the shared KV region work with the provider and the decoder in one process, because that was the
only way the region could travel. This carries it out: the decoder exports, a unix socket carries the
dma-buf fds, a provider in another process maps them as its own KV tensors, and the decoder adopts the cells.

Base branch 1bit/hrx-vulkan-patched (as #163 was).

What it adds

• POST /slots/<id>?action=share {"socket": PATH} — the slot's KV region, exported and sent to PATH as
  {fd, offset, size, layout} per region: one sendmsg carrying a length-prefixed JSON header and
  SCM_RIGHTS. The reply carries the region layout, the context params a provider has to reproduce, and a
  share_id. /proc/<pid>/fd cannot be reopened for a dma-buf on Linux (measured), so a socket is the
  only way the fds reach another process.
• POST /slots/<id>?action=adopt {"tokens": [...], "share_id": N} — cells 0..N-1 become positions 0..N-1 of
  the slot's sequence with no data movement (llama_kv_cells_adopt), and the slot's prompt is set so the next
  request hits that cache. The share_id is required and single-use: without it a client could adopt cells no
  export just produced, and whatever bytes sat there would silently become a prompt's cache. Both actions are
  new server task types (SLOT_SHARE, SLOT_ADOPT) and run on the context thread like save/restore, so a
  share never races a decode.
• llama_kv_share_params() — n_ctx, n_batch, n_ubatch, n_seq_max, flash attention and K/V types. The layout
  hash can only say yes or no; this says what to build. K/V types now live in llama_cparams (they were only
  in llama_memory_params, which does not outlive init). The struct is llama_kv_share_cparams because a
  function of the same name shadows it in every TU that includes llama-ext.h.
• tools/kvshare-prefill — the reference provider, in both transports: --fds prefills into the decoder's
  KV memory, --out writes the sequence state the decoder restores. It is a CPU prefill on purpose: the fd
  plumbing can be gated with no NPU, and the NPU runtime is the same program with its own BO import.

Measured (strixhalo, Qwen3-0.6B-Q4_K_M, ctx 20480, -fa on, greedy 16 tokens)

The decoder is llama-server on HRX0; the provider is llama-kvshare-prefill on the CPU; the engine drives
the handoff. Region 2240 MiB in 3 dma-buf chunks (28 layers × 8 kv heads × 128 × f16 × K+V × 20 480).

prompt tokens	file transport: restore	shared transport: share + adopt	bytes moved by the handoff
251	4 ms	0.43 ms + 0.00 ms	0
3 189	39 ms	0.49 ms + 0.00 ms	0
12 722	198 ms	0.57 ms + 2.00 ms	0

(share is export + the fd handoff, from the engine's stderr line; the file column is the same provider, same
model, same decode, writing a 1.46 GB state file at 12.7k tokens. Server-side export+sendmsg alone is
0.016-0.021 ms.)

The gate: identical greedy text on fd vs file, three prompts at three lengths. That is the transport
check, and it needs the same provider on both sides — which is why kvshare-prefill grew --out. It passes
at 251 / 3 189 / 12 722 tokens. The reply is not always identical to --device hrx: a CPU prefill and an
HRX prefill differ numerically, and the greedy token after a long prompt flips (it matches at 3 189 tokens
and differs at 251 and 12 722 on this model). That is the provider's math, not the handoff — the same CPU
provider through the file route gives the same text as through the fds at every length.

Limits, stated as limits

• The region is llama_kv_cache: a hybrid model's recurrent half is not shared, and ?action=share refuses
  a hybrid cache rather than hand over half a prompt. Verified, not assumed, on two hybrids: Qwen3.8-Flash-Next
  (linear attention) and the 27B itself (bonsai2-unrot-q4_0, 48 GDN layers) — both answer 501 ("no
  shareable region: either the server was not created with ONEBIT_KV_SHARE=1, or the model is hybrid and its
  recurrent state is not shareable"), the engine falls back, and the request is answered identically
  ('Red, Orange, Yellow' / the same 27B answer on every route). 0 gfxhub faults. The folded ternary 27B cannot
  exercise this — it is hrx-only, so 1bit serve refuses the hybrid route before the handoff is reached.
• Both sides must be the same pinned llama.cpp, the same model, and the same -fa/-c/-b/-ub/KV types.
  A mismatch is a refusal (share_import checks the layout hash), not a wrong answer.
• The measured provider is a CPU prefill — a stand-in. The NPU prefill provider is pi-bonsai's lane
  (TBONSAI-SPEED); it maps the same descriptors as XDNA BOs. So these numbers price the handoff, not the
  route: a hybrid still only wins if the provider beats the backend's own prefill, which is the conclusion
  docs/hybrid.md already records.
• A shared handoff that fails says why and the engine forwards the request as it came.

The environment these numbers were taken in

strixhalo, kernel 7.3.0-rc4-1bit-perfopt+, HRX's HSA runtime = TheRock's nightly libhsa
(/opt/rocm-therock/…/_rocm_sdk_core/lib/libhsa-runtime64.so.1, md5 c455cf0e, 5 141 585 B — what
cmake/hrx.cmake bakes in via ONEBIT_HRX_LIBHSA), and every arm with
HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1.

That last part is not cosmetic: on this runtime the SVM path produces gfxhub PERMISSION_FAULTS: 0x8
(UTCL2 client CPF) from llama-server on some runs — pi-attn pinned it on the SVM path, and the CWSR-style
env above is how every HRX arm runs from now on. The gate run quoted above added 0 gfxhub faults, and a
separate 3-run A/B of the env on vs off found no throughput cost worth the name (122.8 vs 125.4 ms median,
spread 120.6-125.7, identical answers): the workaround is free, which is why it is used everywhere rather
than debated.

Re-run at the final heads of this branch (engine 312592c + fork 39a72359) reproduced the transport gate
(PASS) and 0 gfxhub faults; its absolute times are not quoted because the box was at 54 GB used with 14 lanes
loaded at the time, which inflated the file-side restore to 17/80/513 ms. The numbers in the table are from the
run with the die to itself.

Verified against the merged engine

The engine half is in main (engine ggml-org#714, merge 9e1164a). Re-gated this branch on top of it —
engine 4d56634 + fork 39a723593, five routes, no-SVM env, gpu-hold:

prompt tokens	file restore	share	adopt
251	5 ms	0.29 ms	0.00 ms
3 189	56 ms	0.94 ms	1.00 ms
12 722	208 ms	0.56 ms	2.00 ms

transport gate PASS, 0 gfxhub faults. And the plumbing now has a CI-runnable test on the engine side
(engine ggml-org#740): the fake decoder speaks ?action=share/?action=adopt, the fake provider fails unless it was
handed one readable fd per region, 11 checks, no model/GPU/exporter.

[bong-water-water-bong]

VERDICT: CHANGES

Built and exercised at 2742299e9. The plumbing works and fails safely in the cases I could reach without an HRX build; the blocker is a trust boundary the endpoints do not have.

Built: ninja llama-server llama-kvshare-prefill → both link, rc 0 (17 files, +919 −18 over the base).

Exercised on a CPU-only build (so share cannot export; that is the point — it shows the error paths):

call	result
POST /slots/0?action=share with no exportable region	501 This server's KV cache has no shareable region; start it with ONEBIT_KV_SHARE=1 — clear, actionable, no wedge
POST /slots/0?action=adopt {"tokens":[1,2,3]} on a fresh, empty cache	200 {"id_slot":0,"n_adopted":3,...}
same adopt with the 6 ids of a prompt whose cells are already occupied	500 Cannot adopt the shared cells: this slot's cache does not hold the provider's prompt — the occupied-cell guard works
server after both	still serves a correct greedy completion

The blocker: adopt trusts the caller completely, and there is nothing that ties it to a share. On a fresh cache it returned 200 for [1,2,3] — a slot now claiming three positions of KV that no provider ever wrote. In the intended flow the cells are empty at adopt time too (the provider writes tensor data, not cell metadata — that is the whole point of adopt), so "cells are empty" cannot distinguish "the provider filled the region" from "nobody did". Two concrete consequences, both silent:

1. Any local client can set a slot's cached prompt with one adopt call and skip prefill for it, producing an answer computed over uninitialised KV. No authentication, no epoch, no state that a share happened.
2. More importantly for the engine: ggml-org#714 checks n_adopted != ids.size() and treats a match as success. A provider that exits 0 but fills only part of the region yields n_adopted == ids.size() and an engine that believes the handoff worked — so a partially-filled KV produces wrong output with nothing to detect it.

Cheapest fixes, in increasing order of effort: (a) have the provider write a small header into the region (magic, version, token count, and the layout hash) that adopt verifies before marking cells — this is the same idiom the region already carries; (b) at minimum, require that the slot had a successful share since its last decode, and return 4xx otherwise, so a bare adopt cannot fabricate cache state. Also worth one line in the body: the trust boundary is "whoever holds the fds may write the region", and it is currently implicit.

Structure note for merge order: this branch is stacked — it carries #163's +245 −14 and #162's +30 −2 under it (tests/test-backend-ops.cpp, src/llama-kv-share.*, ggml-cpu are the same commits). Merge #161 → #162 → #163 first, then this rebases to its own diff; merging in another order will look like a much larger change than it is.

Not verified, and cannot be here: the measured table and the "identical greedy text on the fd route vs the file route" gate. Both need an HRX decoder exporting dma-bufs; this box has no HRX build after the rebuild. The reference provider compiles and links, and the endpoint error paths behave, which is what this tree can show.

No tprs-ready. The tool (tools/kvshare-prefill, 230 lines) I read but did not run end to end — it is gated behind the same missing HRX exporter.

[bong-water-water-bong]

VERDICT: READY

At 39a72359 (+676 −2, 11 files). My blocker on the previous head was that adopt trusted the caller and nothing tied it to a share; that is fixed, and I re-ran the exact call I used to demonstrate the hole.

Before / after, same request:

previous head:  POST /slots/0?action=adopt {"tokens":[1,2,3]}                      -> 200 {"n_adopted":3}
this head:      POST /slots/0?action=adopt {"tokens":[1,2,3]}                      -> 400 "adopt needs the \"share_id\" that ?action=share returned"
this head:      POST /slots/0?action=adopt {"tokens":[1,2,3],"share_id":12345}     -> 400 "adopt must quote the share_id of the export these cells came from (POST /slots/<id>?action=share first)"

The implementation is the shape it should be: ?action=share mints share_id = ++kv_share_seq and records it per slot (kv_share_issued[id_slot]); ?action=adopt refuses a missing, zero or mismatched id; and on success it does kv_share_issued.erase(issued); // single use, so an id cannot be replayed against a second adopt. Combined with the cells-occupied guard, a slot can now only take cells that the client's own export produced — the two silent-wrong-output paths I described (a bare adopt fabricating cache state, and ggml-org#714's n_adopted == ids.size() check passing over a partially-filled region) both need a share the server issued.

Built at this head: ninja -j4 llama-server llama-kvshare-prefill → both link, rc 0 (40 targets). Endpoints exercised on a CPU-only build, which is enough for the task types and error paths — share still answers 501 "no shareable region; start it with ONEBIT_KV_SHARE=1" when there is no exporter.

The three PRs now interlock, which is worth recording: ggml-org#714 sends the share_id (merged), this validates and consumes it, and ggml-org#740's fake backend tests the contract in CI — so the trust boundary is enforced, and it is enforced by a test that runs on every PR rather than by a device run.

Still not verified by me, unchanged: the measured table and the "identical greedy text on the fd route vs the file route" gate — both need an HRX decoder exporting dma-bufs, and this box has no HRX build. The reference provider (tools/kvshare-prefill) compiles and links.

Merge-order note (unchanged): this branch still stacks #161/#162/#163 under it — merge those first and it rebases to its own diff.

tprs-ready applied.