Activity

1. Anhelone commented on Sep 29, 2026

   

   Anhelone

   on Sep 29, 2026

   More actions

   Hi! I have a feature idea that could be very useful for long-context agent workflows.

   Right now my typical setup looks like this:

   MAIN agent: ~150K context
   ↓
   switch to subagent: ~30–60K context
   ↓
   return to MAIN

   The problem is that when returning to MAIN, a large part of the 150K context may need to be processed again.

   With prompt processing around 1000–1800 tok/s, rebuilding 150K tokens takes roughly:

   1000 tok/s → ~150 sec
   1500 tok/s → ~100 sec
   1800 tok/s → ~83 sec

   So one MAIN → subagent → MAIN switch can cost around 1–2.5 minutes.

   Would it be possible to add an explicit save/restore API for conversation state?

   For example:

   POST /sessions/main/save
   POST /sessions/main/restore

   or something similar to llama.cpp slot save/restore.

   Then an agent system could do:

   MAIN 150K
   ↓
   save MAIN state to SSD

   SUBAGENT 50K
   ↓
   work normally

   restore MAIN state
   ↓
   continue from the same context

   If restoring the saved state from a fast NVMe SSD took, for example, 5–15 seconds instead of reprocessing 80–150 seconds of prompt, that would make switching between agents much faster.

   It would also make it possible to keep several large agent sessions without keeping all of them active in RAM/VRAM:

   MAIN 150K
   Coder agent 80K
   Research agent 60K

   Only the active one would stay in memory, while inactive conversation states could be stored on SSD and restored when needed.

   Ideally this could also survive a Strata restart.

   Maybe the existing RAM conversation cache could remain the fast first level, while saved conversation states on SSD act as a second level.

   This would be especially useful for Hermes, OpenClaw, Cline and other systems where a main agent frequently delegates work to subagents and then resumes its previous long context.

   Would something like persistent conversation-state save/restore fit Strata’s architecture?
2. Niko1221 commented on Sep 29, 2026

   

   Niko1221

   on Sep 29, 2026

   Owner

   More actions

   Strata reuses a checkpoint only when the new prompt starts with exactly the same tokens, and it keeps one conversation. So either OpenClaw changes something early in the prompt (a date or a dynamic block in the system prompt), or other requests (subagents, compaction) run in between and replace the cached conversation. Could you share a few consecutive prompt N tokens = R reused lines from the log? We'll add a log line showing where the prompts first differ; multi-conversation caching and save/restore are tracked in #57.
3. Niko1221 commented on Sep 30, 2026

   

   Niko1221

   on Sep 30, 2026

   Owner

   More actions

   Closing since we haven't heard back. If you can share a few consecutive prompt N tokens = R reused lines, please reopen or open a new issue. Multi-conversation caching is tracked in #57 (and #189 implements it).
4. Niko1221

   closed this as completedon Sep 30, 2026
5. q738531569-tech commented on Oct 27, 2026

   

   q738531569-tech

   on Oct 27, 2026

   More actions

   Found it on my box — and the cause was --conversation-cache-* (parking), not the client.

   I have a 2× V100-16GB box running Qwen3.8-Flash-Next IQ3_XXS at 524288 context. Same symptom as @jinkang06: a long agent session re-reads the whole prompt on most steps. But mine came and went, so I could compare days.

   What I did: for each day, I took every reading the prompt: START of TOTAL line from the journal and computed TOTAL - START — how many tokens actually had to be read. Then I took the median.

   2026-10-23   median    5 tok    parking OFF    (8.9% of requests read >50K)
   2026-10-24   median   31 tok    parking OFF    (5.2%)
   2026-10-25   median   31 tok    partly on      (25.1%)
   2026-10-26   median 69,081 tok parking ON     (55.1%)   ← 43 s of reading per step
   

   That is a 2,000× difference in the same session, same model, same context size. The only parameter that changed is parking.

   The engine log shows what parking does on every single turn:

   strata serve: conversation cache: dropped 1 superseded copy of this conversation; parked=0
   strata serve: conversation cache: parked 289349 tokens in 1543.3 ms; ... reused_kv_bytes=0
   strata serve: prompt 295180 tokens = 4501 reused + 290679 read in ...
   

   So on a plain continuing conversation — one session, nothing else running — parking still copies the outgoing conversation's checkpoints and K/V into host RAM, then the next request brings them back. DETAILS.md calls parking "a switch-time decision", but the engine treats every request as a switch: the conversation that just finished has to be parked before the next one goes in.

   My guess at why that kills reuse: the copy out and the copy back are not free. conversation_checkpoints_split / merge move the checkpoint list around, and a failed merge does checks.clear() (generate.cpp, the MergeBack destructor). Restore then re-validates every checkpoint — it must be an exact token prefix, and "checkpoint image identity differs" / "checkpoint image prefix differs" are hard failures that make the whole snapshot ConversationRestore::invalid. One mismatch and you fall back to a normal full read. Related: #3928 already documented that resume == 0 triggers checks.clear() on the next request.

   With parking off, the same session is fine:

   99%  prompt=331647  read=56      660 ms
   99%  prompt=37129   read=85      678 ms
   99%  prompt=38791   read=52      528 ms
   

   95-99% reuse, prefill 0.5-2.2 s.

   So, two answers for the original question:

   • One conversation at a time → turn parking off. Leaving it on costs you a re-read on every step. This is not obvious from the docs; the flag is described as making multi-conversation switching cheap, with no note that a single continuous conversation pays for it too.
   • Several conversations alternating → parking is worth it, because without it every switch back is a full read. But then avoid --conversation-cache-mib smaller than your snapshots: [Engine] Stop during a long prompt read loses the read prefix; the next message re-reads 375K tokens (22 min) #3928 shows a 6.9 GiB snapshot against a 3.84 GiB budget silently degrading to a full read.

   @Niko1221 — if this matches your model of the code, the useful doc line would be something like: "A single conversation that never switches gains nothing from parking and should leave --conversation-cache-mib at 0; on some boxes each turn pays a full snapshot copy plus a restore validation."

   Box for reference: 2× Tesla V100-SXM2-16GB, --peer-device 1, engine 0.1.47, --max-context 524288 --kv int8 --kv-resident 32768 --prompt-cache 64 --prompt-cache-spread, 125 GB RAM, session 290K-330K tokens. Numbers above come from the normal engine log over four days, not a controlled benchmark.