[oscarlius]

Problem and result

On multi-GPU layer splits, conversation capture, parked restore and batch-slot copies serialize across stages. This PR adds opt-in persistent stage workers and optional reusable host K/V shuttles. All copy switches remain off by default; cold prefill and prompt admission are unchanged.

The engine-code validation anchor is cfff8ec3ec1db4f1226d451cd2f8e73b47aa317f. The evidence-only follow-up adds the standalone ownership probe and sanitized records linked below; engine implementation and defaults are unchanged. It preserves the original PR history, incorporates the StagePool shutdown fix, and merges upstream main 870aedd9e1ebe1c91f94da93b59af17729f6c5c0. The only manual conflict was documentation; both the upstream arena-memory guidance and the copy-switch limits were retained.

Related discussion: #3321, including the maintainer's request for a PR: #3321 (comment) . Earlier approaches #3742 and #3711 were considered.

Approach and boundaries

• STRATA_STAGE_COPY_PARALLEL=1: persistent stage workers for capture, parked restore, direct and host slot copies. Publish a snapshot only after all stages succeed; propagate worker errors and exceptions.
• STRATA_SLOT_SHUTTLE_REUSE=1: reuse storage but copy the full image. Equal token prefixes do not establish equal K/V bytes. This switch remains off by default; the measured reuse arm below regressed short concurrent throughput.
• STRATA_STAGE_COPY_TRACE=1: optional per-stage durations; off by default.
• STRATA_PARK_ASYNC=1 or STRATA_CKPT_ASYNC=1 takes precedence and disables the pool with a visible conflict message. Their shared staging state keeps serial stage copies. Preserve upstream serial checkpoint pool/move/top-up behavior.
• Shutdown sets the stop predicate while holding each worker's wait mutex before notifying. This closes the observed lost-wakeup window. The pool serializes its own shutdown/run paths; callers must still obey normal C++ object lifetime rules. A test that raced object destruction with caller dereference was invalid and was removed; it is not counted as coverage.

Engine-code validation anchor cfff8ec — 2026-10-30

• Linux x86-64, CUDA 13 / sm_89: engine and snapshot-test executables compiled and linked in a network-isolated, GPU-inaccessible container limited to 2 CPUs / 12 GiB. Six selected CPU CTests passed with no failures or skips. Unchanged build objects were reused; changed dependencies and sources were rebuilt. Engine SHA-256: 47e5ebc9cab42cc2253994bd9fdb6220cc746bae098602ad9522b7ca9cb9e848.
• CPU-only configuration (STRATA_ENABLE_CUDA=OFF, HIP/SYCL off): configure, stage_pool_test build and CTest passed.
• Fresh Clang 20.1.2 ThreadSanitizer build of stage_pool_test: 10/10 rounds passed, zero reported warnings, on one host CPU at nice 19. The test covers worker execution, error/exception propagation, closure destruction and repeated construct/run/destruct cycles.
• Environment-blocked attempt retained: the same TSan binary passed the first container round, then its runtime failed because seccomp denied personality(ADDR_NO_RANDOMIZE). This was not counted as a passing round or silently discarded; host execution of that binary produced the 10/10 result above.
• Exact-head GitHub CI passed: title and metal-link. Metal is a compile/link check, not a hardware runtime test. Current main is included and GitHub reports the PR mergeable; it was Draft at the time those checks were recorded.

Reproduction selections (from the recorded CUDA configuration):

cmake --build /work/cmake-build --parallel 2 --target strata stage_pool_test conversation_cache_test conversation_snapshot_test conversation_split_failure_test conversation_memory_test session_save_reclaim_test conv_cache_test
ctest --test-dir /work/cmake-build --output-on-failure --timeout 120 -R '^(stage_pool_test|conversation_cache_test|conversation_split_failure_test|conversation_memory_test|session_save_reclaim_test|conv_cache_test)$'

No model requests, GPU kernels, production restarts or performance workload were run for the cfff8ec update or the evidence-only follow-up. GPU/model tests below belong to earlier revisions; they are not current-head runtime results.

Earlier controlled integration evidence — 2026-10-29/30

These results belong to local validation merge 30d397635a934b9f38a1b0cda34e2489190f413e (PR 71ec493 plus main 46c152a1), engine SHA-256 a6f1ea0a5354da2d7001e598b129726e59279c0125f2f341eca546eb4894b3fc. They precede the shutdown fix and the latest main merge.

• Qwen3.8-Flash-Next IQ3_S, MTP, 4x RTX 4090, layer split 12/24/36, 16 slots, 262144 context, int8 KV. A-F host/direct/batchsync/async-conflict matrix plus official-main versus patched-default-off completed 100 synthetic requests across recorded rounds, with matching inputs, full response text, completion counts and cached-token counts. Recovery probes are separate.
• OFF / parallel / parallel+reuse each completed 26 functional and 64 performance requests. All 19 groups per arm passed accounting and contamination checks; all paired input/output hashes and prompt/cache/output counts matched. Arms ran sequentially with frozen inputs, not interleaved. These are one-host observations, not general speedup guarantees.
• Parallel versus OFF:16-client aggregate output throughput +7.8%;128K repeat TTFT 0.926→0.349s;256K repeat 2.387→0.620s. Cold128K/256K and native decode medians were approximately unchanged. Aggregate output throughput includes waiting and differs from native decode.
• Parallel+reuse16-client throughput was about 6.0% below OFF, so retain its default-off setting. No defaults change.
• 20 injected copy faults covered capture/parked_restore/slot_to/slot_from/slot_direct on four stages. All expected errors were observed; clean-process recovery passed. After a fault, same-process requests returned 503. This does not demonstrate same-process recovery or an operational self-heal.
• Earlier CUDA fixture modes and compute-sanitizer memcheck passed within their documented scopes. Earlier setup/readiness failures came from harness problems (global urlopen replacement and an exclusive marker write); those failed attempts were preserved separately.

Memory investigation — unresolved stability limit

The parallel 16-client soak stopped at the 32 GiB available-memory guard after 494 iterations / 7904 requests. A separate OFF control completed 533 iterations / 8528 requests over one hour. A second candidate soak also hit the guard. The planned two-hour workload and final-reference parity were not completed; do not describe this as a passed long-duration soak.

Follow-up 400-request sequential ramps used the same engine with parallel ON/OFF. Both grew: ON RSS 130.809→194.838GiB, OFF 130.808→200.400GiB. Their first 300-request slopes were 227.849 / 227.846 MiB per request. This contradicts the earlier claim that only parallel copies grow. It does not establish that allocator retention is the root cause, that all memory was freed, or that this PR has no effect under sustained concurrency. The earlier 'not by this PR / identically' attribution is superseded by this bounded result.

Archived engine logs now identify the initial growth: each parked SavedConversation snapshot accounts for approximately 227.8 MiB. Before eviction, cache-accounting increments averaged 227.810 MiB per park in both arms, matching the RSS slopes above. The runs used a 96 GiB cache budget and a 32 GiB free-memory floor. This initial growth was real cached running/checkpoint/K/V state, not evidence by itself of a leak.

After eviction, the cache accounting fell from 69.411 to 19.134 GiB (parallel) and 70.078 to 15.352 GiB (OFF), while RSS ended at the values above. The cache no longer accounts for that historical peak, but these logs lack live allocation/free counters or allocator statistics. Source inspection shows value-owned vectors destroyed on eviction; worker callables are destroyed before run returns. That does not prove where all remaining resident pages belong. Allocator retention and other unaccounted live allocations still cannot be distinguished conclusively from the archived RSS/cache counters. No malloc_trim, debugger attachment or new workload was used to force a diagnosis. The residual post-eviction gap and the incomplete concurrent soak remain open stability limits.

Cache release and passive RSS evidence — 2026-10-30

Evidence follow-up head: 29a84e8108fd3b50627ddef485aec13544326427. The probe was rerun successfully against exact Git source blobs; recorded fingerprints match those blobs.

The committed validation record includes the revision-bound test inventory, successful and failed/blocked attempts, incomplete soak, reproduction commands, CPU probe source, observed output, header fingerprints and sanitized numeric samples.

• A standalone C++20 CPU probe against the unchanged cfff8ec cache headers tracks five 1 MiB allocations in running state, checkpoint, later-stage state and K/V. Evict and clear return all tracked allocations to the allocator. take() instead transfers ownership: cache accounting reaches zero while the external owner still holds 5 MiB, which is freed when that owner is destroyed. Three cases passed. This is a bounded, single-threaded ownership check, not a GPU, concurrency or allocator-to-OS reclamation test.
• Natural-traffic observations came from deployed revision 71ec4933, not the latest PR engine build. UTC 03:14:02→03:15:02: eviction count 0→41, cache 58.615→2.534 GiB, anonymous RSS 108.518→107.566 GiB; in-flight requests 0→1. At 03:23:02 cache had refilled to 46.470 GiB while anonymous RSS was 107.669 GiB and the service was idle.
• Refill of about 43.936 GiB with about 0.103 GiB additional anonymous RSS supports reuse of resident pages. Allocator retention remains an inference, not a confirmed root cause. There are no synchronized allocator free/in-use and live-object counters; concurrent natural activity and ownership transfers confound net byte differences. This does not exclude all unaccounted live allocations or prove the PR unrelated to sustained-memory risk.
• The first 30-minute observation completed 31 samples/22 natural requests but had no eviction. The second captured the event and 10/30/60-second follow-ups, then stopped early on recorder-disk exhaustion: 29 valid samples/~25 minutes, not a completed 30-minute run. No extra traffic, restarts, debugger/probe attachments or malloc_trim were used.

Remaining limits

This PR is now submitted for maintainer review, with the following validation gaps explicitly open. Current-head GPU/layer-split execution, model parity/performance and sustained stability are not rerun under the no-production-restart/no-load constraint. HIP/SYCL builds and Metal runtime execution remain untested here. CI checks must be tied to the displayed revision: the checks linked above belong to cfff8ec; follow-up check results are recorded separately. CONTRIBUTING.md backend build requirements remain incomplete; targeted maintainer help with HIP/SYCL and device runtime validation is requested before promotion. Preserve all failed and environment-blocked attempts; no performance or memory-root-cause claim is inferred from passing CPU tests.

Maintainer review request — 2026-10-30

Please review the ownership, synchronization and failure paths and identify the minimum backend/runtime checks needed before merging. Ready for review is not a claim of merge approval or production acceptance. Allocator retention remains an inference: passive eviction/refill observations and the CPU ownership probe do not uniquely attribute residual RSS or exclude all live allocations. The latest head has not been exercised with GPU/layer-split model parity, performance or sustained stability; HIP/SYCL builds and Metal runtime remain missing. The committed validation record reflects Draft status when recorded; this review request supersedes that status recommendation, not its evidence limits. No merge, production restart or workload is requested.

[oscarlius]

Supplemental v0.2.5 rebase measurements — partial configuration comparison

I reviewed the per-request records from a stock v0.2.5 run and a separately built v0.2.5 candidate carrying this patch. The cached-input improvement is reproducible in these records, but this is not an isolated OFF/ON experiment for #4002, and it does not establish that long-context admission/queueing is solved.

Build and scope

• Base: official v0.2.5 fa663fabc5c2c568d6035b3ae853b38e1762865a; replay of this PR: bfe8f69fdb586dc04fffafa1ce057afee69c330c; candidate: 854e063d823f5e0112acd171c4c729562f64c3f8.
• The candidate additionally contains separate serving/cache changes. They are a confounder. The current PR head at review time remains a2cde216c537c82a6637b63f057c5b1f633e3d23; this comment does not mean the PR branch has been rebased/pushed.
• Candidate executable SHA-256: f37e34b8de19218b1077e155802ac87dd63ef0c1498ca5052b362778fcc29f45, matching its build receipt and live executable. All seven performance files match the recorded build manifest and the candidate Git content after CRLF/LF normalization; they are not byte-identical to Git blobs.
• Qwen3.8-Flash-Next, native engine 0.1.50, four RTX 4090s, 16 serving slots, 262,144 context. Host: dual Xeon Platinum 8380 / 80 physical cores. GPU limits were independently observed at 350 W per card across the sequential deployment period. This is not a matched comparison against older 450 W runs.
• Candidate: STRATA_STAGE_COPY_PARALLEL=1; shuttle reuse and copy tracing disabled. Runs were sequential, not interleaved. No matched candidate OFF arm was recorded.
• Native regression targets: 6 passed, 1 skipped, 0 failed. The skipped target was conversation_snapshot_test; this should not be reported as seven passing tests.

All three recorded rounds (seconds unless indicated; stock → candidate)

Metric	Stock rounds	Candidate rounds	Median comparison
Single-request TTFT	0.427 / 1.458 / 2.231	0.847 / 0.820 / 0.737	1.458 → 0.820; noisy baseline
Single native decode, tok/s	178.3 / 169.2 / 180.6	175.7 / 167.9 / 181.1	178.3 → 175.7; no decode-kernel gain established
Short-prompt 16-client aggregate output, tok/s	260.898 / 255.741 / 254.996	277.401 / 302.850 / 326.451	255.741 → 302.850 (+18.4%)
Short-prompt 16-client TTFT p50	4.384 / 4.438 / 4.449	3.885 / 3.374 / 3.118	4.438 → 3.374 (-24.0%)
Short-prompt 16-client TTFT p95*	8.334 / 8.547 / 8.582	7.653 / 6.741 / 6.006	8.547 → 6.741 (-21.1%)
128K first-input TTFT	12.591 / 12.649 / 12.650	12.329 / 12.416 / 12.477	12.649 → 12.416 (-1.8%)
128K repeated-input TTFT	1.441 / 1.681 / 1.611	0.483 / 0.374 / 0.580	1.611 → 0.483 (-70.0%)
256K first-input TTFT	26.572 / 26.727 / 26.689	26.198 / 26.180 / 29.584†	Do not aggregate as three clean cold pairs
256K repeated-input TTFT	3.077 / 2.983 / 2.925	0.849 / 0.851 / 1.802†	Descriptive only: 2.983 → 0.851 (-71.5%)

*The runner's p95 over 16 requests uses index int(0.95*n), so this is the maximum of the 16, not a well-estimated population tail. Aggregate output throughput includes TTFT and is distinct from native decode speed.

†The candidate's third “256K first” request was not cold: 223,232 cached + 38,517 freshly processed tokens, versus stock 0 cached + 261,749 fresh tokens. Its following repeat is also marked as a sequence caveat. Both rows are retained above rather than silently discarded; no clean three-round 256K-cold median claim is made. The first two cold pairs were approximately unchanged. The 128K cold requests each processed 130,675 fresh tokens with zero cache.

Evidence limits and outstanding work

• Each arm has 19 groups / 64 request records, including warmup. All recorded requests completed with no recorded HTTP/SSE error and nonempty generation. All 64 paired completion-token counts, finish reasons, and retained 120-character output samples match. The one prompt/cache-count mismatch is the contaminated 256K request described above. This is a liveness/performance check, not a semantic/full-output validation or a 26-case functional-suite result.
• Both summaries have null 2K/8K prefill probes. Their subsequently written prefill_results.json files are byte-identical; these cannot establish independent A/B prefill measurements and are excluded. In those files ttft_s also equals native prompt_ms/1000, which must not be presented as separately measured client TTFT.
• The 16-client group uses short prompts. It does not retest sixteen concurrent cold 128K inputs, the historical ~331 s TTFT case, sustained cache-hit queueing, or >120/600 s timeout behavior. No timeout/disconnect occurred in these short recorded requests, but they do not establish those issues as fixed.
• Serving configuration at measurement time had no shared 32,768-token default and no explicit 600-second engine-silence setting (startup default 300 s). These records used explicit short output limits; they do not validate 32,768-token generation or a 600-second live timeout.
• HIP/SYCL/Metal, GPU failure injection, state-transition stress, and sustained production stability remain unvalidated. Keeping the PR Draft is appropriate.

Conclusion: this v0.2.5 candidate retains the observed cache-repeat TTFT benefit, with little change in cold-input latency and no demonstrated native decode gain. Attribution to only #4002, and resolution of the main long-context scheduling bottleneck, still require controlled follow-up evidence.

[oscarlius]

@Niko1221 This PR is now ready for review at 29a84e8108fd3b50627ddef485aec13544326427. Please review staged-copy ownership, synchronization and failure handling, and advise which backend/runtime checks are required before merge.

Allocator retention remains an inference, not a confirmed RSS root cause. The committed CPU ownership probe and passive eviction/refill data do not exclude other live allocations; natural-traffic measurements came from an earlier deployed revision.

The latest head has not completed GPU/layer-split model parity/performance or sustained runtime stability; HIP/SYCL builds and Metal runtime validation also remain missing. These gaps and all incomplete/failed attempts are retained in the PR body and committed validation record. Ready for review does not mean merge approval or production acceptance. No merge or production restart is requested.