[simon3z]

Overview

This PR automates the slots save/restore orchestration to files and life-cycle management (e.g. delete old slots) leveraging the session-id headers sent by most of the clients. This code, besides the added tests, has been tested with Pi ("sendSessionAffinityHeaders": true, "sessionAffinityFormat": "openai") and switches between sessions with different context is automatic and immediate as the prefill phases are avoided by loading the relevant slots from disk.

Additional information

All the additional information about this PR can be found in the issue #29678.

Requirements

No additional requirements introduced.

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - AI (pi agent, qwen3.8-27b) assisted with the implementation, the test additions, and improving the documentation. I reviewed the code and own the design choices. See the Assisted-by line in the commit.

[simon3z]

Hi @ngxson and @ServeurpersoCom, first time contributing, I wonder if you'd be interested in reviewing and merging this enhancement. Let me know if there is something I should do or change to get this approved. Thank you in advance!

[ServeurpersoCom]

I've got a good to-do list; I think I'll be able to take a look at it before the weekend!

[ServeurpersoCom]

Hi, one thing bothers me: the session path is active as soon as one of these headers is present, even without --slot-save-path, so it changes the default behavior for clients that already send them.

In that case slot selection ignores prompt similarity and the evicted slot is no longer saved to the RAM cache, so sub-agents sharing a long system prompt or users switching between conversations get more full re-prefills than today. With --slot-save-path set, each session switch also blocks all slots during a synchronous write that can reach several GB.

Have you tried raising --cache-ram for your multi-agent setup? The existing prompt cache already restores evicted conversations by prefix, and the 8 GiB default may simply be too small for your contexts.

Could the whole feature be opt-in behind an explicit flag, so nothing changes for users who don't enable it?

[simon3z]

Hi, one thing bothers me: the session path is active as soon as one of these headers is present, even without --slot-save-path, so it changes the default behavior for clients that already send them.

Hi @ServeurpersoCom, yes, indeed, I realized I should have addressed that, I prepared a follow up revision (just submitted) where you enable this feature by specifying which headers to use (e.g. --slot-save-sessions x-session-id,session_id,x-session-affinity --slot-save-sessions-max 10).

Have you tried raising --cache-ram for your multi-agent setup? The existing prompt cache already restores evicted conversations by prefix, and the 8 GiB default may simply be too small for your contexts.

Yes, indeed that is helping in some situations, when contexts are fairly similar at the beginning and then fork (e.g. sub-agents), however these are the things that --cache-ram isn't covering (and maybe doesn't even belong there for the amount of space required):

• Resume long-running sessions over an extended period of times (e.g. you stopped the development of something for hours or days, you used the agent for other development, and then you come back to it)
• Entirely different sessions (e.g. working almost in parallel to multiple projects or scripts, etc.)
• Survives across restarts of llama-server and reboots (reduces the "stress" of "oh I need to restart llama-server, etc." because of waiting minutes for the context to be rebuilt)
• Saves power

Maybe the sweet spot is combining both.

Could the whole feature be opt-in behind an explicit flag, so nothing changes for users who don't enable it?

Definitely! Just updated the patch (details in the first paragraph of this comment).

Thank you for the review and your time!

[ServeurpersoCom]

Thanks, the opt-in looks good! Looking closer though, there are quite a few edge cases:

• at startup the old session files get pruned but never added to the sessions map, so the cap doesn't apply to them
• that cleanup runs as soon as --slot-save-path is set, even without the new flag, and fs::remove can throw there
• only the target context is restored, the draft/MTP one keeps the previous slot's KV, so speculative decoding gets much slower after a restore
• no context checkpoints either, so hybrid/SWA models re-prefill everything once the client rewrites history
• in router mode all models share the same save path, same id means same file
• the session id shows up in /slots and in responses, so anyone can grab someone else's session
• saving is synchronous, so every session switch freezes all slots for the whole write
• a couple of -Wsign-compare warnings

Since the RAM prompt cache already handles the draft and the checkpoints, maybe the sweet spot you mentioned is just letting it spill evicted entries to disk? That would cover your use cases without needing a session id at all.

[ServeurpersoCom]

Quick follow-up on my last suggestion: there's already #28092 doing this, but it got too complex. As @ngxson said in his reply there, he has his own version based on chained hashes of token blocks, and he wants to do some refactoring first before implementing the feature. So it's probably worth waiting for that rather than building a third approach in parallel.

[simon3z]

@ServeurpersoCom do you think #28092 is going to be prioritized any time soon?
I have been rebasing/using this patch for a while, it's self-contained and builds on top of an existing feature (save/restore slots). I wonder if it would make sense to get this included while we wait for the more advanced #28092 which seems quite complex to get finalized.
What do you think?