[Engine] dsv41 0.1.51: decode on fresh prompts waits on serial store reads (disk queue < 0.4 on a 990 Pro); DS4_AHEAD=1 is 10% slower (RTX 5090, Windows) #4170

All numbers measured on my machine unless marked "derived" (computed from measured counters).

Setup

• Strata 0.2.6 / Control 0.1.51, engine strata-dsv41 0.1.51, build 562a0c2ed716+dsv41+cuda13.0, NVIDIA driver 617.42, Windows 11.
• Hardware:
  • RTX 5090 32 GB, PCIe 4.0 x16 (board limit).
  • Ryzen 7 7800X3D.
  • 128 GB DDR5 (4 × 32 GB at 3600 MT/s).
  • Samsung 990 Pro 2 TB, PCIe 4.0 x4 (behind the B650 chipset), model on this drive.
• Model: DeepSeek V4.1 Flash 1017, pack ist-das-2.25bpw, context 256k.
• Launch: --set reserve-mib=8192 --set ssd-cache=96 --speed faster (experts 5).
• Engine log at start:
  • RAM store: up to 8436 slots of 11.4 MB (96.0 GB), policy 0, 8 reader threads, unbuffered reads
  • RAM store: 8180 of the profile's hottest experts read in (74.8 GB) in 12.8 s
  • 13906 routed experts on the CPU lane (135.3 GB, in place in the model's files), 8 threads
  • CUDA0 … experts 14.53 GB (1454 of 15360 in VRAM)

Observation

On repeated prompts decode runs at about 25 tok/s; on never-seen prompts it drops to about 15 tok/s. That second case is what an agent workload (Hermes) actually sees. During the slow case the SSD is far from busy.

Method:

• Windows PhysicalDisk counters for the model drive, sampled every 0.7 s.
• Each phase uses only the samples inside its own requests' time window.
• Two baseline runs (A1, A2), each a fresh launch with the same expert profile file.
• Same prompts, temperature 0, no reasoning, 256 tokens per request.

Derived

• Per-read service time. By Little's law (time per read = queue ÷ reads per second), fresh-prompt reads take about 0.30–0.44 ms each for about 1 MB. So roughly one read is in flight at a time.
• Data per token. About 48 MB per token is read from the SSD on fresh prompts, about 5 experts' worth at about 9.7 MB each.
• Time per token. Fresh decode costs about 64 ms per token against about 40 ms on repeated prompts, so about 24 ms more. About 50 reads of about 1 MB at 0.3–0.44 ms each, one after another, is about 15–22 ms. That is most of the gap.
• Unused headroom. The same drive delivers 4.3 GB/s average (5.9 GB/s peak) at queue depth 1.8 during the store preload, against 0.74–0.76 GB/s during fresh decode.

What is not the limit (measured)

• RAM bandwidth.

  • A C# probe (built-in .NET, 4 GB buffer, engine stopped) reads 48–51 GB/s, against 57.6 GB/s theoretical at 3600 MT/s dual channel. Random latency is 104 ns (4 KB pages).
  • On cached decode, a 2-thread memory-read load (38 GB/s) cost about 21%, while a 2-thread CPU-only load cost about 1%. So cached decode is partly RAM-sensitive.
  • Fresh-prompt decode is dominated by the SSD wait above.

• PCIe 4.0 x16 (GPU). nvidia-smi dmon during decode: host→GPU 0.33 GB/s mean (1.3 GB/s peak), about 1–4% of the link, warm and fresh alike. Only a 13.7k-token prompt read uses it heavily: 13.3 GB/s mean, peaks 26–33 GB/s.

• GPU power. Power limit 400 W: decode draws 142–171 W at 2931 MHz, with no power, thermal or hardware throttling.

• More store reader threads. With queue < 0.4 the 8 store readers are mostly idle. I did not test DS4_STORE_THREADS > 8 for that reason.

• Router lookahead into the store (DS4_AHEAD=1, A/B/A, environment of the test process only):

  Run	Repeated prompts	Fresh prompts
  A1	25.0 tok/s	15.1 tok/s
  DS4_AHEAD=1	22.5 tok/s	14.8 tok/s
  A2	24.9 tok/s	16.2 tok/s

  • It reads 20–25% more from disk.
  • Outputs are identical: the text hash matched for all 13 requests in the three runs.
  • This matches your note in init_store() ("measured 4.5 against 4.7 tok/s… DS4_AHEAD=1 turns them on"). Keeping it off is right here too.

Question

On a miss, the decode path seems to wait on the store reads of a layer about one at a time: queue 0.26–0.38 while the drive can sustain 4–6 GB/s at queue about 2.

1. Could the misses of a layer (and/or the 1 MB pieces of one expert) be issued together, so the NVMe queue fills while the CPU lane waits?
2. Is the per-layer dependency the hard limit, so that only better guesses (lookahead) can hide it?
3. Are the st.ahead_* counters (ahead_pf, ahead_hit…) exposed anywhere (log or /metrics)? I could not find where they are printed.

I can rerun any of this with a build or a flag you suggest and post the numbers.