Decode speed is bistable within one session: per-layer host per expert entry jumps 3-5x and stays there, while entries / GPU-reach wait / CPU kernel time stay constant #1913

Environment (short)

• Strata 0.1.42 (tag v0.1.42 commit 61b3fb5, v0.1.40, v0.1.41), CUDA build
• RTX 3080 20 GB (sm_86), single GPU, driver 616.92
• Xeon E5-12C/24T, AVX2 (no AVX-512), Win 10, 96 GB RAM
• Qwen3.8-Flash-Next IQ2_XS, native pack
• Flags: --spec 4 --spec-min-p 0.5 --kv int8 --kv-resident 0 --max-context 262144 --vram-reserve-mib 700 --host-core last --pool-workers 12
• Env: STRATA_DECODE_TIMING=1, STRATA_POOL_SPIN_US=20000, STRATA_PREFILL_CPU_SHARE=0
• Expert arena on 2 MB large pages; route tail skip ON (default)
  (STRATA_PREFILL_CPU_SHARE=0 is deliberate: no AVX-512 here, this is 0.1.41: STRATA_PREFILL_CPU_SHARE on by default halves decode throughput on a CPU without AVX-512 (i9-14900KF + RTX 5090 D, Windows), and auto decides sharing pays when it does not #1729 - per-layer host CPU term 15.7 -> 71.5 ms with the share on.)

1. What happens
   11 consecutive requests inside one engine process, no config change between them:
   4TS fixed 80ts #1-6: 91.4 - 99.8 tok/s, per-layer host 17.9-21.8 ms, entries/window 4.76-5.43, host per entry 3.4-3.9 ms
   serve: fix zero-token replies from engine-queue race between requests #7-11: 20.1 - 44.7 tok/s, per-layer host 61.1-137.2 ms, entries/window 4.99-5.30, host per entry 11.3-15.7 ms
   Between request 6 and 7 the speed drops 98.2 -> 44.7 tok/s and never recovers for the rest of the session.

2. What does NOT change across the same 11 requests

• entries/window: 4.76-5.43 before, 4.99-5.30 after (same work handed to the CPU pool)
• GPU-reach wait: 12.72-13.75 ms throughout (GPU is not the variable)
• CPU kernel time: 3.82-4.24 ms at ~5 entries on both sides
• avg T (draft acceptance): 3.72-3.92 on both sides
• plan / actq / jobs in the [plan actq jobs CPU] breakdown: all <= 0.5 ms in every row
  So the jump is essentially all in the CPU term - the pool dispatch wall clock - not in compute, and not in lock contention inside the engine. The pool takes 3-5x longer to hand back the same number of experts, and stays that way.

3. For scale
   Over ~180 requests on this machine, bucketed by expert-cache hit rate, the normal cost of one missed expert is 3.4-5.1 ms of host time, and speed tracks the hit rate as a step. In section 1 the same machine and build show 11-16 ms per entry - the per-entry cost itself tripled. No amount of cache would have fixed that.

4. Three things I noticed in the same window (may be unrelated)
   (a) The PCIe probe varies a lot between starts on an idle machine: 12.4, 9.0, 9.1 GB/s host->device, with individual samples from 4.5 to 12.4, moving pcie_frac 0.25 -> 0.34. It is the number the new adaptive pcie_frac starts from.
   (b) VRAM is nearly exhausted: "389 MiB of VRAM free with everything loaded", and "strata verify: capturing the 5-token window (235 MiB of VRAM free)" appears in the lines immediately before request 7, the first slow one. Causal or coincidence, I cannot tell - the timing lines up.
   (c) Widening --max-context 131072 -> 262144 costs the expert cache ~17% of its slots (7661 -> 6291) on the same model, since --kv-resident 0 keeps the whole KV resident.

5. Possible bug: --pool-workers 12 and 14 both print only 11 pinned processors
   With --host-core last, every count prints its own number of logical processors except these two:
   11 pool workers on logical processors 0,2,4,6,8,10,12,14,16,18,20 (11 listed)
   12 pool workers on logical processors 0,2,4,6,8,10,12,14,16,18,20 (11 listed) <- 12 requested
   14 pool workers on logical processors 0,2,4,6,8,10,12,14,16,18,20 (11 listed) <- 14 requested
   Workers past the 11th appear to run unpinned. Measured here that shows up as 12 ~= 14 (tied) and 16 clearly worse. Intended, or a placement bug?

6. Side note on tuning
   Before the arena used large pages, --pool-workers 6 beat the default on this machine (37.1 -> 55.0 tok/s), which matches the --calibrate hint. After large pages, 12 is best (91.9 tok/s vs 80.6 at 6). If --calibrate is meant for Windows users, it may be worth having it notice whether the arena is on large pages - following the current guidance here would pick 6, which is 24-31% off the best.

7. Questions

• 1. Could the engine detect and report the section-1 jump? The counters already exist in the decode timing line: a one-line warning when "host per entry" exceeds ~2x its own session median would turn "it feels slow" into "the log says why".
• 2. Is there anything to make the pool resilient to external CPU pressure? plan/actq/jobs stay <= 0.5 ms, so it looks like the workers are simply not being scheduled. Here 11 pinned workers + 1 host occupy 11 of 12 physical cores, so any other CPU consumer lands on top of them. Would leaving a core empty (--pool-workers 10) trade peak speed for stability, or should the spin/priority already handle that?
• 3. Is the section-4(a) probe spread expected (4.5-12.4 GB/s)?
• 4. Section 5 - intended or bug?

8. Method
   Everything above comes from the engine's own STRATA_DECODE_TIMING=1 line, one per request; "host per entry" = the CPU term divided by entries, both from that line. The 11 requests in section 1 are inside one process - the log has one "session is up" and no "engine started" between them. What I cannot show: the wall-clock stamps are inherited from the parent process, so I cannot align them with outside events, and I did not capture a process list while it happened. External CPU preemption is my first guess (the same signature appeared here before right after Windows Defender scanned a freshly downloaded engine), but I would rather report the numbers than the guess.

Happy to split sections 4-6 into separate issues if preferred.