[midhatn]

An agent using one local model for both its primary conversation and auxiliary approval/summary calls can lose its primary KV state between turns. The ordinary conversation checkpoints do not preserve an independent positional KV cache, so returning from an unrelated side call can reread thousands of tokens.

This adds an opt-in, bounded host-RAM snapshot around explicitly marked auxiliary requests. A primary → auxiliary → primary sequence restores the primary only when the token/image prefix and control-vector setting match; rewritten history falls back to normal prefill. The feature defaults off and does not add parallel model instances.

Changes

• --conversation-cache-mib N parks one complete mutable primary state, including QSA device/host KV, GDN recurrence/convolution, PLE history, MTP state and the associated checkpoints/token/image metadata. Model weights and expert placement stay in place; device addresses remain stable.
• The API adapter forwards a literal strata_auxiliary: true as aux=1 for text and image requests. Further marked auxiliaries do not displace the parked primary, even if their prompts are longer. Reasoning level and prompt text are not used to infer auxiliary intent.
• Admission enforces the byte cap and a 2,560 MiB available-physical-RAM floor immediately after allocation. Allocation/copy failure falls back to cold prefill; partial restore failure terminates the engine rather than continuing with inconsistent state.
• The adapter prefers the running engine's INFO version to an adjacent BUILD.json. This fixes a custom 0.1.13 executable being displayed as 0.1.12 when it shares a directory with a retained official fallback. Older engines still use the manifest fallback.
• Add API/protocol tests, a standalone real-CUDA/host round-trip test with checks active in Release, and usage/validation documentation in docs/AUXILIARY_STATE_CACHE.md.

Validation

Based on upstream b89c989a7155e984e544ddd90d1038dda7da9e3d, including the prompt-cache correctness fixes after 0.1.13. The existing patch was also compatibility-tested against v0.1.16; the published implementation has not yet been rebased onto that release.

Hardware used for bounded validation: Ryzen 9 7940HS, RTX 4070 Laptop 8 GB, and 64 GB RAM, running Windows.

• Server/adapter tests and the release engine build passed.
• The native host/device snapshot test passed round-trip restoration, replacement, admission bounds and invalidation checks.
• Bounded checks covered same-image restoration, changed-image invalidation, an auxiliary longer than its primary, rewritten-history invalidation, and subsequent matching-prefix restoration.
• Code outputs were independently executed and checked; tool-call, reasoning and auxiliary-request routing checks also passed.
• A long-context known-answer continuation after marked side calls crossed the resident KV window and preserved the expected answer while reusing the primary prefix.

These are bounded checks, not the requested repeated streamed-KV soak or proof of arbitrary agent-task success. They do not establish bitwise logit parity, cross-platform correctness or a general generation-speed improvement.

Scope for review

This is a draft for review of the API and state ownership. Physical-memory admission currently uses Windows telemetry; other platforms decline admission safely. Linux/cgroup-aware admission is not implemented or validated. The snapshot cap excludes existing small checkpoints, and its RAM-floor check is not a continuous system-wide reservation. Bitwise logit parity across all model/configuration combinations is not claimed.

The cache protects one primary around explicitly marked side calls. It is not persistent storage, arbitrary multi-session caching or a solution for rewritten compression prefixes. The auxiliary inference itself still runs. Deployment-specific configuration and agent delegation changes are outside this PR.

Related to the prompt-cache work in #39, but this snapshots the complete mutable state around explicit auxiliary calls rather than changing checkpoint retention or telemetry. No maintainer action or merging is assumed.

[Niko1221]

Thanks - this is careful work, and the state it saves is complete (QSA KV with the streaming maps and host copies, GDN, PLE history, MTP, the checkpoints); off by default it changes nothing. Before merging I'd like:

1. A parked side call that resumes from live keeps resume (it is reset to 0 now, so every such call reads its prompt from scratch).
2. Size the snapshot by the conversation, not by --max-context: it copies the whole pools and host KV, so at 128K-262K it is 3-6 GB whatever the conversation's length. Copy the pages below live.size().
3. Count the parked checkpoints against the cap, and skip the zero-fill of the vector.
4. Linux admission (MemAvailable from /proc/meminfo) - without it the feature never engages on Linux - and a warning when --prompt-cache 0 disables it; the 2,560 MiB floor as a flag.
5. The GEN/GENI protocol comments and the server docstring updated; guard n_qsa_layers() == 0.
6. A soak with KV streaming that checks the answer text across primary -> side call -> primary. Please rebase on main after 0.1.14 (the stdin reader and the PCIe path changed).

[midhatn]

Thanks for the detailed review. The six points remain outstanding in this draft; compatibility testing does not resolve those implementation requests.

The existing snapshot and version-reporting commits were tested against v0.1.16 (c0107c3), including the 0.1.14 stdin/PCIe changes and 0.1.15 eager CUDA loading. Both commits cherry-picked cleanly and the rebuilt engine passed bounded checks. The published implementation has not been rebased onto v0.1.16.

Remaining work

1. Preserve a valid live resume point for a continuing auxiliary request.
2. Capture the conversation's required pages rather than maximum-context pools.
3. Include checkpoint storage in the cap and avoid unnecessary buffer zero-initialization.
4. Add Linux MemAvailable admission, make the physical-RAM floor configurable, and warn when --prompt-cache 0 disables the feature.
5. Update protocol documentation and guard zero-QSA-layer configurations.
6. Rebase onto current main and run the requested repeated primary → auxiliary → primary soak with streamed KV and answer-text comparisons.

Bounded compatibility checks

Test hardware: Ryzen 9 7940HS, RTX 4070 Laptop 8 GB, and 64 GB RAM, running Windows.

Server/adapter checks, native snapshot round-trip/admission/invalidation checks, image-prefix checks, rewritten-history checks, independently executed code outputs and auxiliary-request routing checks passed. A known-answer continuation crossing the resident KV window restored the primary prefix after marked side calls and returned the expected answer.

These checks do not validate a combined multi-conversation/NVMe design, replace a soak, or establish Linux correctness, bitwise logit parity or a generation-speed improvement. Rewritten compressed history still needs fresh prefill wherever its prefix no longer matches.

The release's memory behavior should be measured before reusing an older deployment's tuning. CUDA allocator accounting and global/NVML free memory are different measurements; neither should be substituted for the other without explanation.

This PR remains a draft pending the requested changes and validation.

[midhatn]

Following the convergence discussion in #57, I posted a concrete shared snapshot design and acceptance proposal here:
#57 (comment)

It proposes one capture/restore core, automatic bounded RAM prefix caching, an optional disk tier, physical-memory admission and correctness gates covering streamed KV, images, failures and real agent workflows. This is a proposal for review, not a completed implementation or agreement on ownership.

Keeping #41 in draft as a reference while that design is discussed. The latest commit only revises validation documentation; it does not change runtime code or rebase the implementation.

[Niko1221]

Thanks @midhatn - since the #57 core covers this and you're happy for it to be superseded, I'll close it once the unified PR is open. Your fix that reports the engine's own version (INFO engine=) instead of BUILD.json's is still needed on main: I'm taking it into 0.1.20 with credit to you.

[midhatn]

Thanks, that works for me. Happy for #41 to close once the unified PR is open, and thanks for carrying the version-reporting fix into 0.1.20. I'll focus on the Windows memory checks and agent tests for the shared core.

[Niko1221]

Thanks @midhatn. The version fix shipped with credit to you, and the snapshot work carries on in #57's shared core, where your Windows admission tests are already helping. Closing this in favour of that.

[midhatn]

Thanks for carrying the version fix upstream and for the credit. I've updated to 0.1.27 and will keep the snapshot work with #57 rather than reopen this. The small local auxiliary-cache patch still passes the bounded Windows checks; I've put the current results in #57.