Repository navigation
parallel:1 / 0.1.42 -- whole prompt re-read 22x in one session; any total-length change kills the prefix #1930
Description
Summary
Windows, 2x RTX 3080 20 GB layer split, engine 0.1.42, "parallel": 1 (single-slot path), Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS + MTP, --ple-io ram, STRATA_ROUTE_TAIL_SKIP=0. One real agent conversation grew from ~122K to ~192K tokens over one session.
The whole prompt is re-read from token 0 on almost every turn. In one session the engine logged 22 full re-reads (each "reading the prompt" sequence starting at 8,192 and climbing to the full length), i.e. a ~190K-token context re-computed 22 times. At ~195 s per full read that is ~71 minutes of pure reading in one session.
Two things make this worth a look:
-
"expert cache hit" does not describe the KV prefix. Every turn -- full re-read or not -- reports "expert cache 92-96% hit", i.e. the same number. A turn that re-read 190K tokens took 216 s; a turn that read 5 tokens took 20 s; both say ~93% hit. So the one "cache" number the log shows is the MoE expert-weight hit rate, not the prompt prefix. It cannot be used to tell whether a re-read happened.
-
The 0.1.42 single-slot log line dropped the "reused N" field. In the parallel:2 reports (
--batch 2 --batch-groups autoon a real agent client: 9 full prompt re-reads (608 s of reading) in 23 minutes, sibling group decode falls to 4-13 tok/s #1906) the line read "prompt 54301 tokens = 0 reused + 54301 read in 38786 ms" -- you could see a full re-read at a glance. In 0.1.42 / parallel:1 it is only "reading the prompt: 8,192 of 190,854 tokens, 5 s so far", with no reuse count. Whether the prefix matched is not observable from the log.
Rig, config
- Windows 11 / WDDM, 2x RTX 3080 20 GB, layer split auto
- Xeon E5-2696 v3 (18C/36T), no AVX-512, 128 GB RAM
- engine 0.1.42 (strata.exe 165,004,288 B)
- "parallel": 1, --ple-io ram, STRATA_ROUTE_TAIL_SKIP=0, --kv int8
- client: agent conversation on a LAN machine, one conversation, one turn at a time
Evidence 1 -- full re-reads vs incremental turns, same session, same config
Both kinds of turn are in the same log. The only visible difference is the read duration:
| line | generated | duration | result line |
|---|---|---|---|
| 2512 | 889 tok | 20 s | done, expert cache 92.9% hit |
| 2490 | 860 tok | 216 s | done, expert cache 92.7% hit |
| 2446 | 471 tok | 207 s | done, expert cache 96.5% hit |
| 2407 | 840 tok | 216 s | done, expert cache 95.3% hit |
| 2689 | 841 tok | 19 s | done, expert cache 95.9% hit |
| 2612 | 821 tok | 27 s | done, expert cache 93.0% hit |
The 20 s turns read only a handful of new tokens (their "reading the prompt" line is "194,856 of 194,861", i.e. +5 tokens). The 216 s turns re-read the whole ~190K. Same expert cache hit rate, 10x duration difference.
Evidence 2 -- every re-read is preceded by a change in prompt total length
In time order, the prompt total at each of the 22 re-reads:
122,816
130,768 +7,952
128,402 -2,366
136,553 +8,151
135,085 -1,468
139,867 +4,782
136,923 -2,944
139,947 +3,024
160,756 +20,809
154,359 -6,397
155,831 +1,472
161,849 +6,018
175,449 +13,600
172,197 -3,252
176,032 +3,835
189,935 +13,903
188,227 -1,708
193,093 +4,866
189,470 -3,623
193,967 +4,497
190,854 -3,113
192,401 +1,547
Every total differs from the previous one (up or down), and every one triggers a full read. There is no case of "same length, still re-read" nor "length changed, still reused". The prefix cache is only kept alive when the length happens to be identical and the prefix tokens are unchanged -- which for this agent is almost never.
The agent appends tool results (up) and trims/compacts old context (down) every turn, and on hand-off re-concatenates the message list, so the prompt head is not stable turn to turn.
Evidence 3 -- 22 re-read start lines (all at 8,192)
Log line numbers of every "reading the prompt: 8,192 of ...":
28, 113, 268, 413, 460, 486, 532, 855, 1044, 1212, 1322,
1428, 1592, 1959, 2054, 2255, 2324, 2366, 2413, 2448, 2693, 2736
Each is followed by a contiguous rising sequence (8,192 -> 16,384 -> 24,576 -> ...) with no return to 0, i.e. a single read of the entire prompt, not a loop. 22 such sequences = 22 whole-prompt recomputes.
Suggestion
-
Log: print the KV-prefix reuse count ("N reused + M read") on the single-slot path as well, and rename the expert number so it is not confused with the prompt cache. Right now "expert cache 93% hit" next to a 216 s turn reads like "the cache worked" -- it did not.
-
Engine: when the new prompt differs from the previous only near the tail, try to reuse the longest common token prefix instead of discarding the whole context on any length change. If the client's prompt is only appended to, this alone would cut the ~195 s reads to near zero.
-
This is distinct from the
--batch 2 --batch-groups autoon a real agent client: 9 full prompt re-reads (608 s of reading) in 23 minutes, sibling group decode falls to 4-13 tok/s #1906 report (parallel:2, "reused 0" on hand-off). Here it is parallel:1 and the trigger is any total-length change, not concurrency.
Raw evidence
- Log: E:\Strata\server-console.log, lines 28-2737
- Engine: Strata v0.1.42
Activity
ylzbj1-stack commentedon Oct 11, 2026
Follow-up on the reuse answer in the batch A/B/C commit ("reuse works, up to 96% of a turn's prompt reused, 1286/1341").
That measurement was taken on a 1,341-token prompt. The #1930 case is a 190,854-token prompt -- about 140x larger. A small prompt fitting in the prefix cache says nothing about the large case, because the cost is not linear in the missed fraction once the prefix breaks: one full re-read of 190K tokens is ~195 s of reading (see the log table in #1930), and in that session it happened 22 times.
Could you re-test the reuse rate on a 190K-class prompt whose total length changes by a few hundred tokens between turns (the exact pattern in #1930, Evidence 2)? If reuse holds there too, then the 22 re-reads in my log must come from something else and I would like to know what. If reuse collapses to 0 on the length change, that is the confirmation.
For reference, in this 0.1.42 / parallel:1 log the single-slot path no longer prints "N reused + M read" -- the only way to see a full re-read is to spot a "reading the prompt: 8,192 of ..." sequence. So I cannot give you the reused/read split for the 22 events myself; only the read durations (216 s vs 20 s) are visible.
CullenShane commentedon Oct 11, 2026
I'm running into this issue with
replicas: 2andparallel: 1as well. It was working great until I got a prompt that was 260k+ and then all requests had no more kv cache anymore, and I started experiencing what appear to be race conditions between two requests, where a request that had generated roughly 10% of its token limit was stopped for producing too many tokens, while at the same time the other request was prompt processing.While the bugs coupled with the key value cache are troubling, the real improvement would be a log message that says the key value cache is exhausted and more sane behavior about storing the prefixes that can be stored rather than just giving up and re-processing everything once the prompt cache has filled.
What if I want to expand my prompt cache to accommodate the larger requests? The message should tell me how to do that.
ylzbj1-stack commentedon Oct 11, 2026
Correction to my own analysis -- the trigger is more specific than "any total-length change". I found the actual head-field that moves.
The OpenAI request line logs msgs, tools and prompt_tokens. In this session the tools count alternates 79 <-> 83 while a conversation runs:
The client is a local agent with a "background review" pass. During a review pass it mounts 4 extra tools (83 = 79 + 4) and unmounts them afterwards. The tool definitions sit at the very front of the prompt, so a tools count of 79 -> 83 -> 79 breaks the KV prefix at the head -- the entire 130K-190K prompt after it is recomputed, every time.
This is the same root cause as the 260K report from CullenShane ("no more kv cache anymore" / "prompt cache has filled"): it is not the KV cache capacity that is short (context is 262144 here and the prompt is 190K, it fits). It is that the head of the prompt is not stable turn to turn, so nothing after the changing field can be reused.
So this is a client-side issue, not an engine bug: the engine is right to recompute when the prefix token sequence differs. But two things would help a lot:
And one engine-side ask stands: on the parallel:1 / 0.1.42 log line, print the reused/read split again (as parallel:2 does). Without it, "tools 79<->83" is only visible because I looked at the req line, not at the read line.
ylzbj1-stack commentedon Oct 12, 2026
Follow-up: the same 0.1.42 single-slot session, with the share of wall clock the re-reads cost
Same rig as the original report, engine 0.1.42, single-slot path. This run is on the abliterated IQ3_S (not the IQ3_XXS of the original post), so it cannot be confused with the 0.1.41 session:
auto(CUDA0 layers 0-24, CUDA1 25-47, one hand-off per window)"parallel": 1,--batch-groups auto(no--batch),--max-context 262144 --kv int8 --kv-resident 32768 --prefill auto --spec 4 --spec-min-p 0.5 --ple-io ramWhat the session shows
38 prompt reads, 8 of them with
reused 0. Those 8 are 1,065 s of the 1,380 s of total read time — 77% of all prompt-read wall clock.The worst single turn is 5 min 9 s of reading for one request.
Two rows that are not "the prompt is just long"
1. A 164-token prompt took 4.1 s (39.6 tok/s). If the cost were the re-read itself, this row would be free. Something is holding the read, not computing it.
2. The read speed is bimodal, not size-dependent. 182,101 tokens read at 2,679 tok/s; 219,597 tokens read at 712 tok/s — same class of prompt, 3.8x apart. The same split shows up in the partial-reuse reads: 201,681 tokens (
reused 16,384) at 2,735 tok/s, against 126,522 tokens (reused 16,384) at 652 tok/s and 169.0 s, and 217,215 tokens (reused 212,833) at 371.8 tok/s. The slow reads are not the long ones.That second point looks like the shape in #1913 (a 3-5x, bistable, within-session slowdown while the per-expert entry counts stay flat), so I am reporting it here rather than claiming it is the same bug.
What this adds
The original post showed that
reused 0happens on almost every turn. This session prices it: 77% of all prompt-read time goes to reads that reuse nothing, on a workload of two long agent conversations. The head-field change is still the only thing I can find that moves between consecutive turns of the same conversation; if atools-count change is enough to drop a 130K-220K prefix even when the conversation body is byte-identical, that alone accounts for every row above.Niko1221 commentedon Oct 19, 2026
Thanks for taking the time to report this.
We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to read properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.
To reopen it takes about two minutes:
/reopen. A check runs and either reopens the issue or lists what is still missing.The rules are in CONTRIBUTING.md.