[wannagoxuf]

Problem and result: on a layer split, the copies that move a conversation - the capture into the conversation cache and the shuttle between the sessions and the batch slots - ran one stage after another, although every stage owns its own card, its own sessions and its own PCIe bus. The other cards sat idle while one stage faulted its whole share into host memory. With the stages running at the same time, one worker per GPU, the capture of a ~110K-token conversation costs 260 ms per 100K tokens instead of 848 ms (medians, same box, same workload, the same binary with the new switch off as the baseline).

The branch is rebased on the current main (7dd5cf1, release 0.2.4.2, engine 0.1.49.2). The change itself is unchanged by the rebase - the patch's +/- lines are byte-for-byte the same as before it - and the numbers below were measured again on the rebased branch.

Related discussion: #3321 (the layer split's copy path), #3711 (the page-reuse part of the same series - this PR is the concurrent-copy part it does not cover).

Approach: a small persistent pool (include/strata/core/stage_pool.hpp), one worker per stage, runs the per-stage bodies of the three copy paths: the cache capture, the slot shuttle's device-to-device path and its round trip through host memory. The stages are independent - each owns its sessions, its device and its private copy stream - so no lock is needed, and every call site runs on the engine's serial orchestration thread.

The workers are persistent on purpose: the snapshot copies ride a private non-blocking CUDA stream that is thread-local (conversation_copy.hpp), so a thread per copy would leak a stream per stage per copy.

Compatibility: what is copied does not change, only when. A stage that fails does not stop the others (their parts are discarded with the image) and the first failing stage's error is reported, as the serial loop's was. STRATA_STAGE_COPY_PARALLEL=0 keeps the serial path. With a single card (no layer split) the pool is not created and nothing changes.

Validation:

Passed:

• Build: cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=89 -DGGML_LLAMAFILE=OFF -DSTRATA_BUILD_TESTS=ON -DSTRATA_ENABLE_CUDA=ON + cmake --build . -j 40 (CUDA 12.8, driver 580.178.04, 4x RTX 4090).

• ctest -j 4 on a quiet GPU (the engine stopped, so the tests can get VRAM): 165 tests, 5 failed. A build of the unmodified main (7dd5cf1), configured the same way, gives 164 tests, the same 5 failed. The failure sets are identical - no new failure and none fixed:

  • ple_parity - the Q2_0 GGUF the test wants is not in the tree
  • expert_parity, pool_test - pack/full/experts.bin is not in the tree
  • platform_memory_test - mlock failed (raise ulimit -l), the machine's limit
  • isa_guard_check - strata::kernels::cpu::iq512x_rows holds wide-ISA instructions and is not in the allowed list (an upstream CPU kernel, the same on the unmodified main)
    The new stage_pool_test passes (0.20 sec).
    The GPU has to be quiet for this comparison: with the engine resident (it holds 89 of the 96 GiB) the parity tests fail with cudaSetDevice: out of memory, and the set of failures then differs from run to run in both builds.

• Output identity: the same five greedy prompts (temperature 0) give byte-identical answers with the switch on and off (sha256 11ec9bcdfe56aabf, 19cce29f43a13012, f666ca2994b41a8a, b5eca0e5ee74e8dd, f785b50574b67e59).

• Before/after, same box and workload: Qwen3.8-flash IQ3_XXS, 4-way layer split 13,26,38, --batch 4, --kv k8v4 --kv-resident 524288, --conversation-cache-mib 393216 --conversation-cache-slots 192, 9 concurrent clients x 3 rounds of ~110K-token conversations. The baseline is the same binary with STRATA_STAGE_COPY_PARALLEL=0. Per 100K tokens (n, median, mean, p90, max; the two runs interleave differently, so the park counts differ - the fresh-capture group is the same work in both and is the cleanest comparison):

  | capture into the cache (fresh) | n=49 | 848 | 785 | 916 | 1163 ms | -> | n=36 | 261 | 255 | 276 | 307 ms |
  | capture into the cache (K/V reused) | n=4 | 403 | 405 | 413 | 422 ms | -> | n=17 | 136 | 130 | 150 | 156 ms |
  | slot shuttle | n=77 | 93 | 257 | 642 | 672 ms | -> | n=64 | 64 | 101 | 285 | 329 ms |

  The capture is 3.3x faster, the slot shuttle's tail collapses (p90 642 -> 285 ms).

Failed: none observed.

Not run: the HIP and SYCL builds - no AMD or Intel toolchain on this machine. The change is portable C++ (the existing OnDevice RAII plus std::thread), but those two backends are not compiled here; a build of them would be welcome before the review.

Limits: the gain needs a layer split (more than one stage). The copies move the same bytes through the same paths, so the copy is now bounded by the slowest stage and by the host memory bandwidth (the round trip through host memory) instead of by the sum of the stages.