[Piggidragon]

A split input that is a window over a KV cache split into streams is one range per stream: the ranges sit a whole kv_size apart and the cells between one range and the next are never read by the graph. ggml_backend_tensor_copy moves ggml_nbytes(), which is the span the ranges are cut from, so the ordered copy moved every gap along with them.

At -c 32768, a 2,048-token window per sequence:

-npl	cells read	cells copied	ratio
2	4,352	18,560	4.26x
4	8,704	26,752	3.07x
8	17,408	30,848	1.77x

This copies the ranges instead, with the 2D copy the backends already have (ggml_backend_tensor_set_2d_async, already public and already implemented by CUDA). It stays blocking, exactly as the copy it replaces was blocking. A unified cache and a single sequence are one flat range and keep the path they had.

Measurements

RTX 4070 (sm_89), driver 610.57.04 / CUDA 13.3, i5-13400F, Qwen3.8-27B-UD-IQ2_M.gguf, -ngl 99 -sm none -mg 0 -t 3 -fa on -ctk q8_0 -ctv q8_0 -b 512 -ub 512 -c 32768, host residency -nkvo --kv-cpu-pinned --recurrent-state-offload, everything under taskset -c 0,2,4. llama-batched-bench -npp 2048 -ntg 128, generation t/s, one process per cell:

-npl	-no-kvu before	-no-kvu after	gain	-kvu before	-kvu after
2	34.12	51.73	+51.6%	51.51	51.71
4	49.51	71.71	+44.8%	71.91	71.98
8	70.91	84.31	+18.9%	83.42	83.51

A cache split into streams reads the same cells as a unified one, and the whole gap between those two columns was the gaps being copied. It closes: after this, the two caches measure the same, and the predicted ratios above track the measured gains.

A unified cache is unaffected to within 0.4%, which is what a change that only touches multi-range windows should look like.

Validation

• Byte-identical output. llama-parallel, 8 concurrent sequences of 16 over a non-unified cache at --temp 0, which seeds its client schedule so the batches repeat run to run. Transcript hash 17f946c340db110b before and after. Its clients ask different questions, which is what makes it a gate: with one prompt shared by every sequence the streams hold the same bytes and a cross-stream read is invisible.

• test_ordered_multi_stream_ranges, added here. It builds the window in the shape attention hands the scheduler (permuted, rows on dimension 1, streams on 3), fills the host store with a long-period pattern and the split backend's buffer with a sentinel, drives this copy path, and then asserts three things: exactly n_stream * used bytes moved, each stream's range arrived whole from its own source offset, and the cells between one range and the next still hold the sentinel. On the parent commit it aborts on the first of those - the copy moves 704 bytes where the graph reads 512 - and it passes here.

  The dummy backend gets an opt-in real_memory flag for it, so the buffers it hands out are backed by memory a test can look at. It is off by default, so the existing tests keep the synthetic addresses they assert on, and all 24 of them still pass.

• test-alloc passes in Release against a CUDA build, 25/25.

• git diff --check clean.

Notes

• The gap bytes are left untouched in the destination copy. Nothing reads them: they lie between one stream's window and the next, outside every stream's n_kv, and attention indexes the copy through its own ne/nb.
• The predicate is deliberately conservative and bails to the existing flat copy unless the last dimension really describes evenly spaced ranges that fit within their own stride, and the source is host memory.
• This does not touch device-to-device copies, weights (the used-experts path above it), or user inputs.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_01JycLdWs6KRgizbdnNAfZqM

[GenerelSchwerz]

Automated preliminary review by OpenAI Codex, for head 6ae4781d51e4.

Blocking: this changes the ordered scheduler copy path but adds no focused regression test. The existing test-alloc pass does not verify the new behavior. Please add a test that constructs a multi-stream host view with sentinel-filled gaps, drives this non-transport copy path, and verifies that every live range reaches the destination while the gaps remain untouched. It should fail on the parent commit and pass with this change.