Crash sometimes serving IQ3_S #2192

strata-vision: model files loaded in 4.4 s
strata-vision: warmed up at 1024 image tokens in 0.9 s
[strata] 2026-10-13 06:17:10 engine started: C:\Strata-0.1.43\engine\strata.exe --serve --pack C:\Strata-data\packs\iq3_s --native C:\Strata-data\models\IQ3_S\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf C:\Strata-data\models\IQ3_S\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile C:\Strata-0.1.43\data\expert-profile.bin --expert-cache auto --prefill auto --spec 4 --mtp C:\Strata-data\mtp\rt --max-context 262144 --kv int8 --kv-resident 32768 --vision --vram-reserve-mib 700 --control-vector-scaled C:\Strata-0.1.43\data\experimental-speed-projection\Qwen3.8-Flash-Next-experimental-speed-projection.gguf:1.0 --control-vector-layer-range 4 44 --cvec-mode project --cvec-dir per-layer --pcie-frac 0.55 --spec-min-p 0.70 --pool-workers 17 --batch 4 --layer-split auto
strata generate: memory guard: limit 511.9 GiB (source: meminfo), current 408.9 GiB, reclaimable cache credited 0.0 GiB; available 103.0 GiB (MemAvailable 103.0 GiB), floor 2560 MiB
strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: C:\Strata-data\packs\iq3_s experts (largest blob 2.66 MB), token embedding IQ4_XS in mapped host memory (322 MiB, 2.0 s)
strata generate: 1466 MiB of weights loaded from C:\Strata-data\packs\iq3_s in 1.1 s (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 2018.88 MiB of weights, in 11.1 s
strata generate: control vector mode = project, dir = per-layer, layers 4..44 (41 steered)
strata generate: PLE direct read probe: 25156 rows/s (fine)
strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off)
strata generate: layer split: CUDA1 holds its weights; 17.15 GiB free (its session follows the split search)
strata generate: profile C:\Strata-0.1.43\data\expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: layer split auto: CUDA0 68 SMs at 1.54 GHz -> 0.69 ms per layer, 16.47 GiB free before its session carve
strata generate: layer split auto: CUDA1 68 SMs at 1.54 GHz -> 0.69 ms per layer, 15.40 GiB free before its session carve
strata generate: layer split auto: CUDA0 link 5.9 GB/s -> miss cost x3.37
strata generate: layer split auto: CUDA1 link 11.4 GB/s -> miss cost x1.75
strata generate: layer split auto: K=18 - predicted 37.6 ms per decode window; the caches hold 16487 of 24576 profiled pairs (~98.7% of the routed mass), best of 46 placements
strata generate: layer split auto: NEAR TIE - K=21 is predicted 37.8 ms per decode window, within 0.7% of K=18 (37.6 ms); the cards differ, so the prediction may not rank them (it prices decode, not the prompt). To try the other: --layer-split 21
strata generate: KV streaming: 32768 of 262144 cells per QSA layer in VRAM, the K/V in 1.03 GiB of pinned RAM
strata generate: PLE on, table 320001536 rows of C:\Strata-data\models\IQ3_S\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf
strata generate: layer split: CUDA1 holds its weights, session [18, 48) and the head; 16.02 GiB free
strata generate: --batch-groups auto: 2 groups of 2 slots
strata generate: --batch: 4 slot sessions on CUDA0 (0.37 GiB each); 15.30 GiB free
strata generate: --batch: 4 slot sessions on CUDA1 (0.65 GiB each); 13.42 GiB free
strata generate: --batch 4: the slot sessions take 1.48 GiB of VRAM on CUDA0 that the expert cache would otherwise hold
strata mtp: draft layer loaded, 835 MiB of VRAM (experts 675, dense 111), files read in 0.94 s (835 MiB/s)
strata generate: GPU 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5
strata generate: multi-GPU under WDDM: at most 8 GiB of the expert arena is pinned (STRATA_ARENA_PIN_GIB changes it)
strata generate: expert arena read unbuffered (0 of 16 probe reads from the file cache; 39.3 GiB available, 51.1 GiB of files)
strata generate: expert arena: large pages are resident without a lock; cudaHostRegister limited to 8 GiB by the engine (multi-GPU under WDDM, or remote experts); 8 slices pinned (7 GiB); large pages (2097152 B)
strata generate: loaded 46.84 GiB at 0.43 GiB/s
strata generate: CPU pool tasks/phase: 54 (automatic), participating threads: 18
strata generate: 17 pool workers on logical processors 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17, the host thread on 0 (draining too) (--host-core first)
strata aux cpus: not available on this system
strata generate: expert cache auto: 15.30 GiB free, 700 MiB reserved (+0 MiB for the draft head) -> 5894 slots
strata generate: only 0 MiB free once the slots are written (reserve 700 MiB); shrinking the expert cache
strata generate: only 0 MiB free once the slots are written (reserve 700 MiB); shrinking the expert cache
strata generate: expert cache 7059 slots, 12.61 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
                 so a reply can differ slightly from a run without the cache (same quality:
                 bench/results/2026-09-27-cache-parity).
                 PROFILE, ranked by routing frequency, no eviction.
strata generate: pre-filled 7059 of 7059 slots from the profile in 3.2 s (4272 MB/s); slot 0 verified
strata generate: layer split, CUDA1: 12.60 GiB free of 22.00, room for experts 11.82 GiB
strata generate: layer split: CUDA1 runs layers 18-47, expert cache 5993 slots (11.82 GiB), 5993 of its 15360 profiled pairs; slot 0 verified
strata generate: layer split: CUDA0 runs layers 0-17
strata generate: layer split auto: L2 probe: the split is startable on the live free figures
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 17 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.43)
strata generate: token graph hit path: 13052 resident experts, decided on the device
strata serve: Windows commit: 8.6 GiB left of a 527.9 GiB limit (RAM + page file)
strata serve: prompt chunk auto: 8192 tokens, a 96-slot ring
strata serve: the prompt path borrows 2034 CUDA0 cache slots (3.66 GiB)
strata serve:   CUDA1 prompt path borrows 1865 of its 5993 slots (3.66 GiB)
strata hc: CUDA1: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata verify: window up to 6 tokens, 75.8 MiB of device buffers
strata serve: layer split: layers 0-17 (CUDA0), 18-47 (CUDA1), one hand-off per window
strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata verify: window up to 6 tokens, 75.8 MiB of device buffers
strata mtp: draft head over 106299 tokens (212.9 MiB)
strata verify: batch windows of up to 4 sequences (layers [0, 18))
strata verify: batch windows of up to 4 sequences (layers [18, 48))
strata serve: 740 MiB of VRAM free with everything loaded
strata verify: capturing the 4-token window (671 MiB of VRAM free)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: capturing the 4-token window (153 MiB of VRAM free)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: capturing the 1-token window (607 MiB of VRAM free)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: capturing the 1-token window (82 MiB of VRAM free)
strata verify: captured the 1-token window (upload no error, sync no error)
strata serve: prompt 270 tokens = 0 reused + 270 read in 3386 ms (79.7 tok/s), 1 generated in 67 ms (14.9 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 82.7% (396 hits / 479 lookups); 1 more read by the GPU over PCIe or from another GPU (0.2% of all 480 routed)
strata serve: KV streaming: 79.82% of 1348 block reads hit VRAM, 1.1 MiB read from RAM
strata batch: slot 0 takes 271 tokens (copied in 163.3 ms)
strata serve: prompt 271 tokens = 270 reused + 1 read in 0 ms (3369.3 tok/s), 1 generated in 25 ms (40.7 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 88.8% (426 hits / 480 lookups)
strata serve: KV streaming: 83.21% of 1620 block reads hit VRAM, 1.1 MiB read from RAM
strata verify: captured the batch window over slots 0
strata verify: captured the batch window over slots 0
strata verify: capturing the 6-token window (633 MiB of VRAM free)
strata verify: captured the 6-token window (upload no error, sync no error)
strata verify: capturing the 6-token window (52 MiB of VRAM free)
strata verify: captured the 6-token window (upload no error, sync no error)
strata batch: slot 2 takes 6474 tokens (copied in 266.3 ms)
strata serve: prompt 6474 tokens = 0 reused + 6474 read in 11100 ms (583.2 tok/s), 1 generated in 26 ms (38.3 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: decode expert cache hit rate: 84.8% (407 hits / 480 lookups)
strata serve: KV streaming: 89.28% of 34920 block reads hit VRAM, 15.1 MiB read from RAM
strata verify: captured the batch window over slots 2
strata verify: captured the batch window over slots 2
strata batch (pipelined, 2 groups of 2): 323 group-steps, 323 rows in 20871 ms = 15.5 rows/s (admissions included)
strata batch: slot 2 gave back 6636 tokens of this conversation (all it holds) in 219.6 ms
strata verify: capturing the 2-token window (574 MiB of VRAM free)
strata verify: captured the 2-token window (upload no error, sync no error)
strata verify: capturing the 2-token window (137 MiB of VRAM free)
strata verify: captured the 2-token window (upload no error, sync no error)
strata verify: capturing the 3-token window (634 MiB of VRAM free)
strata verify: captured the 3-token window (upload no error, sync no error)
strata verify: capturing the 3-token window (77 MiB of VRAM free)
strata verify: captured the 3-token window (upload no error, sync no error)
strata serve: prompt 6637 tokens = 6636 reused + 1 read in 220 ms (4.5 tok/s), 41 generated in 1256 ms (32.7 tok/s), drafts accepted 20 of 26, 0 checkpoints
strata serve: decode expert cache hit rate: 86.6% (19496 hits / 22502 lookups); 58 more read by the GPU over PCIe or from another GPU (0.3% of all 22560 routed)
strata serve: KV streaming: 95.14% of 96548 block reads hit VRAM, 18.9 MiB read from RAM
strata batch: slot 0 takes 6678 tokens (copied in 230.0 ms)
strata serve: prompt 6678 tokens = 6677 reused + 1 read in 1 ms (940.6 tok/s), 1 generated in 27 ms (37.5 tok/s), drafts accepted 0 of 0, 0 checkpoints
strata serve: decode expert cache hit rate: 89.1% (426 hits / 478 lookups); 2 more read by the GPU over PCIe or from another GPU (0.4% of all 480 routed)
strata serve: KV streaming: 95.22% of 98604 block reads hit VRAM, 19.0 MiB read from RAM
strata batch: slot 2 takes 266 tokens (copied in 166.4 ms)
strata serve: prompt 266 tokens = 0 reused + 266 read in 2560 ms (103.9 tok/s), 1 generated in 24 ms (42.4 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 83.3% (399 hits / 479 lookups); 1 more read by the GPU over PCIe or from another GPU (0.2% of all 480 routed)
strata serve: KV streaming: 79.82% of 1328 block reads hit VRAM, 1.1 MiB read from RAM
strata batch: slot 1 takes 6468 tokens (copied in 258.3 ms)
strata serve: prompt 6468 tokens = 0 reused + 6468 read in 9669 ms (668.9 tok/s), 1 generated in 29 ms (34.0 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: decode expert cache hit rate: 85.2% (409 hits / 480 lookups)
strata serve: KV streaming: 85.71% of 22592 block reads hit VRAM, 13.0 MiB read from RAM
strata verify: captured the batch window over slots 0,1
strata verify: captured the batch window over slots 0,1
strata batch: slot 1 takes 257 tokens (copied in 172.9 ms)
strata serve: prompt 257 tokens = 0 reused + 257 read in 2581 ms (99.6 tok/s), 1 generated in 72 ms (14.0 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 84.3% (404 hits / 479 lookups); 1 more read by the GPU over PCIe or from another GPU (0.2% of all 480 routed)
strata serve: KV streaming: 79.75% of 1284 block reads hit VRAM, 1.0 MiB read from RAM
strata batch (pipelined, 2 groups of 2): 402 group-steps, 685 rows in 30144 ms = 22.7 rows/s (admissions included)
strata batch: slot 2 takes 203 tokens (copied in 171.1 ms)
strata serve: prompt 203 tokens = 0 reused + 203 read in 2337 ms (86.9 tok/s), 1 generated in 25 ms (39.6 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 89.6% (430 hits / 480 lookups)
strata serve: KV streaming: 79.84% of 1012 block reads hit VRAM, 0.8 MiB read from RAM
strata batch: slot 0 gave back 6970 tokens of this conversation (all it holds) in 216.1 ms
strata batch: slot 0 takes 6971 tokens (copied in 202.0 ms)
strata serve: prompt 6971 tokens = 6970 reused + 1 read in 217 ms (4.6 tok/s), 1 generated in 36 ms (27.8 tok/s), drafts accepted 0 of 0, 0 checkpoints
strata serve: decode expert cache hit rate: 80.8% (388 hits / 480 lookups)
strata serve: KV streaming: 0.00% of 2052 block reads hit VRAM, 8.3 MiB read from RAM
strata batch (pipelined, 2 groups of 2): 200 group-steps, 200 rows in 4978 ms = 40.2 rows/s (admissions included)
strata batch: slot 2 gave back 304 tokens of this conversation (all it holds) in 114.9 ms
strata serve: prompt 305 tokens = 304 reused + 1 read in 115 ms (8.7 tok/s), 1 generated in 32 ms (31.5 tok/s), drafts accepted 0 of 0, 0 checkpoints
strata serve: decode expert cache hit rate: 80.4% (386 hits / 480 lookups)
strata serve: KV streaming: 0.00% of 308 block reads hit VRAM, 1.2 MiB read from RAM
strata batch: slot 0 takes 306 tokens (copied in 117.2 ms)
strata serve: prompt 306 tokens = 305 reused + 1 read in 0 ms (3473.4 tok/s), 1 generated in 79 ms (12.6 tok/s), drafts accepted 0 of 0, 0 checkpoints
strata serve: decode expert cache hit rate: 91.0% (436 hits / 479 lookups); 1 more read by the GPU over PCIe or from another GPU (0.2% of all 480 routed)
strata serve: KV streaming: 50.00% of 616 block reads hit VRAM, 1.2 MiB read from RAM
strata batch: slot 2 takes 538 tokens (copied in 182.4 ms)
strata serve: prompt 538 tokens = 0 reused + 538 read in 3025 ms (177.9 tok/s), 1 generated in 29 ms (34.1 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 90.0% (432 hits / 480 lookups)
strata serve: KV streaming: 79.91% of 2688 block reads hit VRAM, 2.2 MiB read from RAM
strata batch (pipelined, 2 groups of 2): 31 group-steps, 31 rows in 3871 ms = 8.0 rows/s (admissions included)
strata batch: slot 2 gave back 554 tokens of this conversation (all it holds) in 125.3 ms
strata serve: prompt 555 tokens = 554 reused + 1 read in 126 ms (8.0 tok/s), 329 generated in 6442 ms (51.1 tok/s), drafts accepted 175 of 238, 0 checkpoints
strata serve: decode expert cache hit rate: 95.5% (180862 hits / 189390 lookups); 210 more read by the GPU over PCIe or from another GPU (0.1% of all 189600 routed)
strata serve: KV streaming: 99.69% of 284580 block reads hit VRAM, 3.6 MiB read from RAM
strata serve: suffix drafts: 3 windows, 4 of 9 drafts accepted
strata serve: prompt 484 tokens = 0 reused + 484 read in 2664 ms (181.7 tok/s), 328 generated in 6536 ms (50.2 tok/s), drafts accepted 187 of 247, 1 checkpoints
strata serve: decode expert cache hit rate: 94.2% (176716 hits / 187521 lookups); 159 more read by the GPU over PCIe or from another GPU (0.1% of all 187680 routed)
strata serve: KV streaming: 99.68% of 255088 block reads hit VRAM, 3.3 MiB read from RAM
strata serve: suffix drafts: 7 windows, 15 of 17 drafts accepted
strata serve: prompt 303 tokens = 0 reused + 303 read in 3424 ms (88.5 tok/s), 1 generated in 34 ms (29.4 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 94.4% (453 hits / 480 lookups)
strata serve: KV streaming: 79.89% of 1512 block reads hit VRAM, 1.2 MiB read from RAM
strata batch: slot 0 takes 304 tokens (copied in 171.5 ms)
strata serve: prompt 304 tokens = 303 reused + 1 read in 0 ms (2330.5 tok/s), 1 generated in 86 ms (11.7 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 98.1% (471 hits / 480 lookups)
strata serve: KV streaming: 83.26% of 1816 block reads hit VRAM, 1.2 MiB read from RAM
strata batch: slot 2 takes 6505 tokens (copied in 253.6 ms)
strata serve: prompt 6505 tokens = 0 reused + 6505 read in 10444 ms (622.9 tok/s), 1 generated in 25 ms (40.0 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: decode expert cache hit rate: 95.6% (459 hits / 480 lookups)
strata serve: KV streaming: 95.42% of 98592 block reads hit VRAM, 18.2 MiB read from RAM
strata batch (pipelined, 2 groups of 2): 337 group-steps, 337 rows in 19276 ms = 17.5 rows/s (admissions included)
strata batch: slot 2 gave back 6674 tokens of this conversation (all it holds) in 215.0 ms
strata serve: prompt 6675 tokens = 6674 reused + 1 read in 216 ms (4.6 tok/s), 117 generated in 3076 ms (38.0 tok/s), drafts accepted 59 of 74, 0 checkpoints
strata serve: decode expert cache hit rate: 89.6% (58016 hits / 64727 lookups); 73 more read by the GPU over PCIe or from another GPU (0.1% of all 64800 routed)
strata serve: KV streaming: 97.95% of 277284 block reads hit VRAM, 22.9 MiB read from RAM
strata serve: suffix drafts: 1 windows, 2 of 2 drafts accepted
strata batch: slot 2 gave back 6500 tokens of this conversation (its turn checkpoint) in 165.6 ms
strata serve: prompt 6598 tokens = 6500 reused + 98 read in 1778 ms (55.1 tok/s), 1233 generated in 28303 ms (43.6 tok/s), drafts accepted 490 of 717, 2 checkpoints
strata serve: decode expert cache hit rate: 96.6% (677464 hits / 701135 lookups); 145 more read by the GPU over PCIe or from another GPU (0.0% of all 701280 routed)
strata serve: KV streaming: 99.76% of 3009116 block reads hit VRAM, 29.6 MiB read from RAM
strata serve: suffix drafts: 5 windows, 8 of 11 drafts accepted
strata serve: prompt 927 tokens = 0 reused + 927 read in 3682 ms (251.7 tok/s), 470 generated in 9118 ms (51.5 tok/s), drafts accepted 260 of 325, 1 checkpoints
strata serve: decode expert cache hit rate: 95.0% (245265 hits / 258128 lookups); 112 more read by the GPU over PCIe or from another GPU (0.0% of all 258240 routed)
strata serve: KV streaming: 99.78% of 628284 block reads hit VRAM, 5.6 MiB read from RAM
strata serve: suffix drafts: 2 windows, 3 of 7 drafts accepted
strata serve: prompt 873 tokens = 0 reused + 873 read in 3523 ms (247.8 tok/s), 263 generated in 5233 ms (50.3 tok/s), drafts accepted 138 of 167, 1 checkpoints
strata serve: decode expert cache hit rate: 94.9% (132940 hits / 140108 lookups); 52 more read by the GPU over PCIe or from another GPU (0.0% of all 140160 routed)
strata serve: KV streaming: 99.62% of 297436 block reads hit VRAM, 4.6 MiB read from RAM
strata serve: prompt 276 tokens = 0 reused + 276 read in 3202 ms (86.2 tok/s), 1 generated in 27 ms (36.7 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 96.7% (464 hits / 480 lookups)
strata serve: KV streaming: 79.94% of 1376 block reads hit VRAM, 1.1 MiB read from RAM
strata batch: slot 0 takes 277 tokens (copied in 172.6 ms)
strata serve: prompt 277 tokens = 276 reused + 1 read in 0 ms (2887.7 tok/s), 1 generated in 21 ms (46.5 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 98.1% (471 hits / 480 lookups)
strata serve: KV streaming: 83.09% of 1656 block reads hit VRAM, 1.1 MiB read from RAM
strata batch: slot 2 takes 6478 tokens (copied in 251.4 ms)
strata serve: prompt 6478 tokens = 0 reused + 6478 read in 9823 ms (659.5 tok/s), 1 generated in 24 ms (41.8 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: decode expert cache hit rate: 95.6% (459 hits / 480 lookups)
strata serve: KV streaming: 90.88% of 43136 block reads hit VRAM, 15.9 MiB read from RAM
strata batch (pipelined, 2 groups of 2): 343 group-steps, 343 rows in 17793 ms = 19.3 rows/s (admissions included)
strata batch: slot 2 gave back 6650 tokens of this conversation (all it holds) in 211.7 ms
strata serve: prompt 6651 tokens = 6650 reused + 1 read in 213 ms (4.7 tok/s), 704 generated in 16893 ms (41.7 tok/s), drafts accepted 247 of 382, 0 checkpoints
strata serve: decode expert cache hit rate: 96.9% (391621 hits / 404111 lookups); 49 more read by the GPU over PCIe or from another GPU (0.0% of all 404160 routed)
strata serve: KV streaming: 99.60% of 1729488 block reads hit VRAM, 27.6 MiB read from RAM
strata serve: suffix drafts: 1 windows, 1 of 2 drafts accepted
strata serve: prompt 764 tokens = 0 reused + 764 read in 3186 ms (239.8 tok/s), 398 generated in 8024 ms (49.6 tok/s), drafts accepted 190 of 247, 1 checkpoints
strata serve: decode expert cache hit rate: 96.6% (210907 hits / 218349 lookups); 51 more read by the GPU over PCIe or from another GPU (0.0% of all 218400 routed)
strata serve: KV streaming: 99.74% of 441852 block reads hit VRAM, 4.7 MiB read from RAM
strata serve: suffix drafts: 4 windows, 5 of 8 drafts accepted
strata verify: capturing the 5-token window (589 MiB of VRAM free)
strata verify: captured the 5-token window (upload no error, sync no error)
strata verify: capturing the 5-token window (57 MiB of VRAM free)
strata verify: captured the 5-token window (upload no error, sync no error)
strata serve: prompt 710 tokens = 0 reused + 710 read in 2995 ms (237.1 tok/s), 380 generated in 7717 ms (49.2 tok/s), drafts accepted 230 of 290, 1 checkpoints
strata serve: decode expert cache hit rate: 93.5% (198792 hits / 212524 lookups); 116 more read by the GPU over PCIe or from another GPU (0.1% of all 212640 routed)
strata serve: KV streaming: 99.73% of 399868 block reads hit VRAM, 4.4 MiB read from RAM
strata serve: suffix drafts: 5 windows, 12 of 15 drafts accepted
strata serve: prompt 7345 tokens = 0 reused + 7345 read in 10820 ms (678.8 tok/s), 1 generated in 28 ms (35.3 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 94.6% (454 hits / 480 lookups)
strata serve: KV streaming: 70.51% of 10272 block reads hit VRAM, 12.2 MiB read from RAM
strata batch: slot 0 takes 7346 tokens (copied in 266.1 ms)
strata serve: prompt 7346 tokens = 7345 reused + 1 read in 1 ms (1131.3 tok/s), 1 generated in 24 ms (42.0 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 98.3% (472 hits / 480 lookups)
strata serve: KV streaming: 74.31% of 12328 block reads hit VRAM, 12.8 MiB read from RAM
strata batch: slot 2 takes 13548 tokens (copied in 377.4 ms)
strata serve: prompt 13548 tokens = 0 reused + 13548 read in 19848 ms (682.6 tok/s), 1 generated in 24 ms (41.2 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: decode expert cache hit rate: 92.5% (444 hits / 480 lookups)
strata serve: KV streaming: 68.54% of 10268 block reads hit VRAM, 13.0 MiB read from RAM
strata batch (pipelined, 2 groups of 2): 381 group-steps, 381 rows in 29574 ms = 12.9 rows/s (admissions included)
strata batch: slot 2 gave back 13739 tokens of this conversation (all it holds) in 314.8 ms
strata serve: prompt 13740 tokens = 13739 reused + 1 read in 317 ms (3.2 tok/s), 836 generated in 17865 ms (46.8 tok/s), drafts accepted 463 of 604, 0 checkpoints
strata serve: decode expert cache hit rate: 95.1% (446242 hits / 469452 lookups); 468 more read by the GPU over PCIe or from another GPU (0.1% of all 469920 routed)
strata serve: KV streaming: 99.41% of 2010860 block reads hit VRAM, 47.7 MiB read from RAM
strata serve: suffix drafts: 1 windows, 1 of 5 drafts accepted
strata serve: prompt 8097 tokens = 0 reused + 8097 read in 10601 ms (763.8 tok/s), 566 generated in 11090 ms (51.0 tok/s), drafts accepted 325 of 400, 1 checkpoints
strata serve: decode expert cache hit rate: 95.0% (293638 hits / 309040 lookups); 80 more read by the GPU over PCIe or from another GPU (0.0% of all 309120 routed)
strata serve: KV streaming: 99.47% of 1330980 block reads hit VRAM, 28.2 MiB read from RAM
strata serve: suffix drafts: 3 windows, 1 of 9 drafts accepted
strata serve: prompt 8043 tokens = 0 reused + 8043 read in 10526 ms (764.1 tok/s), 394 generated in 7827 ms (50.3 tok/s), drafts accepted 233 of 298, 1 checkpoints
strata serve: decode expert cache hit rate: 94.7% (208661 hits / 220252 lookups); 68 more read by the GPU over PCIe or from another GPU (0.0% of all 220320 routed)
strata serve: KV streaming: 99.31% of 951020 block reads hit VRAM, 26.5 MiB read from RAM
strata serve: suffix drafts: 1 windows, 1 of 1 drafts accepted
strata serve: prompt 236 tokens = 0 reused + 236 read in 2863 ms (82.4 tok/s), 1 generated in 67 ms (14.9 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 94.0% (451 hits / 480 lookups)
strata serve: KV streaming: 79.93% of 1176 block reads hit VRAM, 1.0 MiB read from RAM
strata batch: slot 0 takes 237 tokens (copied in 169.2 ms)
strata serve: prompt 237 tokens = 236 reused + 1 read in 1 ms (1992.0 tok/s), 1 generated in 30 ms (33.4 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 95.6% (459 hits / 480 lookups)
strata serve: KV streaming: 83.05% of 1416 block reads hit VRAM, 1.0 MiB read from RAM
strata batch: slot 2 takes 18592 tokens (copied in 418.2 ms)
strata serve: prompt 18592 tokens = 0 reused + 18592 read in 24840 ms (748.5 tok/s), 1 generated in 23 ms (43.0 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: decode expert cache hit rate: 95.8% (460 hits / 480 lookups)
strata serve: KV streaming: 66.37% of 10268 block reads hit VRAM, 13.9 MiB read from RAM
strata batch (pipelined, 2 groups of 2): 5 group-steps, 5 rows in 25421 ms = 0.2 rows/s (admissions included)
strata batch: slot 2 gave back 18595 tokens of this conversation (all it holds) in 366.7 ms
strata serve: prompt 18596 tokens = 18595 reused + 1 read in 369 ms (2.7 tok/s), 36 generated in 1102 ms (32.7 tok/s), drafts accepted 18 of 29, 0 checkpoints
strata serve: decode expert cache hit rate: 91.5% (20647 hits / 22560 lookups)
strata serve: KV streaming: 93.08% of 96536 block reads hit VRAM, 26.9 MiB read from RAM
strata serve: prompt 317 tokens = 0 reused + 317 read in 3197 ms (99.2 tok/s), 1 generated in 31 ms (32.3 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 99.0% (475 hits / 480 lookups)
strata serve: KV streaming: 79.80% of 1584 block reads hit VRAM, 1.3 MiB read from RAM
strata batch: slot 0 takes 318 tokens (copied in 167.2 ms)
strata serve: prompt 318 tokens = 317 reused + 1 read in 0 ms (3441.2 tok/s), 1 generated in 25 ms (39.2 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 98.8% (474 hits / 480 lookups)
strata serve: KV streaming: 83.19% of 1904 block reads hit VRAM, 1.3 MiB read from RAM
strata batch: slot 2 takes 19133 tokens (copied in 408.2 ms)
strata serve: prompt 19133 tokens = 0 reused + 19133 read in 22145 ms (864.0 tok/s), 1 generated in 23 ms (43.9 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: decode expert cache hit rate: 99.0% (475 hits / 480 lookups)
strata serve: KV streaming: 63.97% of 10272 block reads hit VRAM, 14.9 MiB read from RAM
strata batch (pipelined, 2 groups of 2): 5 group-steps, 5 rows in 22707 ms = 0.2 rows/s (admissions included)
strata batch: slot 2 gave back 19136 tokens of this conversation (all it holds) in 363.9 ms
strata serve: prompt 19137 tokens = 19136 reused + 1 read in 366 ms (2.7 tok/s), 90 generated in 2579 ms (34.9 tok/s), drafts accepted 57 of 66, 0 checkpoints
strata serve: decode expert cache hit rate: 86.8% (42375 hits / 48845 lookups); 115 more read by the GPU over PCIe or from another GPU (0.2% of all 48960 routed)
strata serve: KV streaming: 95.58% of 209508 block reads hit VRAM, 37.3 MiB read from RAM
strata serve: suffix drafts: 1 windows, 2 of 2 drafts accepted
strata serve: prompt 22006 tokens = 0 reused + 22006 read in 25215 ms (872.7 tok/s), 134 generated in 2971 ms (45.1 tok/s), drafts accepted 79 of 105, 2 checkpoints
strata serve: decode expert cache hit rate: 91.6% (71587 hits / 78130 lookups); 110 more read by the GPU over PCIe or from another GPU (0.1% of all 78240 routed)
strata serve: KV streaming: 96.78% of 343016 block reads hit VRAM, 44.5 MiB read from RAM
strata serve: suffix drafts: 6 windows, 20 of 20 drafts accepted
strata serve: prompt 22182 tokens = 22001 reused + 181 read in 2225 ms (81.3 tok/s), 219 generated in 4608 ms (47.5 tok/s), drafts accepted 141 of 170, 3 checkpoints
strata serve: decode expert cache hit rate: 89.9% (106878 hits / 118837 lookups); 203 more read by the GPU over PCIe or from another GPU (0.2% of all 119040 routed)
strata serve: KV streaming: 98.42% of 860628 block reads hit VRAM, 54.6 MiB read from RAM
strata serve: suffix drafts: 6 windows, 12 of 14 drafts accepted
strata serve: prompt 22504 tokens = 22177 reused + 327 read in 3047 ms (107.3 tok/s), 298 generated in 6135 ms (48.6 tok/s), drafts accepted 198 of 238, 4 checkpoints
strata serve: decode expert cache hit rate: 92.4% (150186 hits / 162566 lookups); 154 more read by the GPU over PCIe or from another GPU (0.1% of all 162720 routed)
strata serve: KV streaming: 99.04% of 1565148 block reads hit VRAM, 60.8 MiB read from RAM
strata serve: suffix drafts: 12 windows, 34 of 36 drafts accepted
strata serve: prompt 22807 tokens = 22499 reused + 308 read in 2720 ms (113.2 tok/s), 59 generated in 1435 ms (41.1 tok/s), drafts accepted 36 of 44, 5 checkpoints
strata serve: decode expert cache hit rate: 90.8% (30059 hits / 33101 lookups); 19 more read by the GPU over PCIe or from another GPU (0.1% of all 33120 routed)
strata serve: KV streaming: 99.11% of 1715088 block reads hit VRAM, 61.4 MiB read from RAM
strata serve: suffix drafts: 5 windows, 10 of 14 drafts accepted
strata serve: prompt 31332 tokens = 22802 reused + 8530 read in 14145 ms (603.0 tok/s), 3116 generated in 72354 ms (43.1 tok/s), drafts accepted 1498 of 2076, 6 checkpoints
strata serve: decode expert cache hit rate: 94.4% (1673342 hits / 1772398 lookups); 722 more read by the GPU over PCIe or from another GPU (0.0% of all 1773120 routed)
strata serve: KV streaming: 99.70% of 9310856 block reads hit VRAM, 113.0 MiB read from RAM
strata serve: suffix drafts: 19 windows, 22 of 47 drafts accepted
strata batch: slot 0 takes 34448 tokens (copied in 655.4 ms)
strata serve: prompt 34448 tokens = 34447 reused + 1 read in 3 ms (291.4 tok/s), 1 generated in 58 ms (17.2 tok/s), drafts accepted 0 of 0, 6 checkpoints
strata serve: decode expert cache hit rate: 94.0% (451 hits / 480 lookups)
strata serve: KV streaming: 99.70% of 9312908 block reads hit VRAM, 113.0 MiB read from RAM
strata batch: slot 2 takes 6474 tokens (copied in 244.7 ms)
strata serve: prompt 6474 tokens = 0 reused + 6474 read in 9772 ms (662.5 tok/s), 1 generated in 43 ms (23.2 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: decode expert cache hit rate: 69.9% (334 hits / 478 lookups); 2 more read by the GPU over PCIe or from another GPU (0.4% of all 480 routed)
strata serve: KV streaming: 89.62% of 34920 block reads hit VRAM, 14.6 MiB read from RAM
strata batch: slot 1 takes 272 tokens (copied in 162.7 ms)
strata serve: prompt 272 tokens = 0 reused + 272 read in 2660 ms (102.2 tok/s), 1 generated in 35 ms (28.9 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 64.5% (307 hits / 476 lookups); 4 more read by the GPU over PCIe or from another GPU (0.8% of all 480 routed)
strata serve: KV streaming: 79.94% of 1356 block reads hit VRAM, 1.1 MiB read from RAM
strata batch (pipelined, 2 groups of 2): 1572 group-steps, 1728 rows in 56020 ms = 30.8 rows/s (admissions included)
strata batch: slot 0 gave back 35235 tokens of this conversation (all it holds) in 562.1 ms
strata serve: prompt 35236 tokens = 35235 reused + 1 read in 566 ms (1.8 tok/s), 1 generated in 37 ms (27.3 tok/s), drafts accepted 0 of 0, 0 checkpoints
strata serve: decode expert cache hit rate: 68.3% (328 hits / 480 lookups)
strata serve: KV streaming: 0.00% of 2052 block reads hit VRAM, 8.3 MiB read from RAM
strata batch: slot 0 takes 35237 tokens (copied in 565.2 ms)
strata serve: prompt 35237 tokens = 35236 reused + 1 read in 3 ms (302.6 tok/s), 1 generated in 76 ms (13.1 tok/s), drafts accepted 0 of 0, 0 checkpoints
strata serve: decode expert cache hit rate: 72.5% (348 hits / 480 lookups)
strata serve: KV streaming: 31.04% of 4108 block reads hit VRAM, 11.4 MiB read from RAM
strata batch: slot 2 takes 835 tokens (copied in 171.5 ms)
strata serve: prompt 835 tokens = 0 reused + 835 read in 3768 ms (221.6 tok/s), 1 generated in 34 ms (29.8 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 65.5% (313 hits / 478 lookups); 2 more read by the GPU over PCIe or from another GPU (0.4% of all 480 routed)
strata serve: KV streaming: 79.96% of 4172 block reads hit VRAM, 3.4 MiB read from RAM
strata batch (pipelined, 2 groups of 2): 792 group-steps, 792 rows in 23339 ms = 33.9 rows/s (admissions included)
strata batch: slot 0 gave back 35634 tokens of this conversation (all it holds) in 561.1 ms
strata serve: prompt 35635 tokens = 35634 reused + 1 read in 565 ms (1.8 tok/s), 1 generated in 36 ms (27.4 tok/s), drafts accepted 0 of 0, 0 checkpoints
strata serve: decode expert cache hit rate: 82.5% (395 hits / 479 lookups); 1 more read by the GPU over PCIe or from another GPU (0.2% of all 480 routed)
strata serve: KV streaming: 0.00% of 2052 block reads hit VRAM, 8.3 MiB read from RAM
strata batch: slot 0 takes 35636 tokens (copied in 551.9 ms)
strata serve: prompt 35636 tokens = 35635 reused + 1 read in 3 ms (306.2 tok/s), 1 generated in 35 ms (28.8 tok/s), drafts accepted 0 of 0, 0 checkpoints
strata serve: decode expert cache hit rate: 78.9% (377 hits / 478 lookups); 2 more read by the GPU over PCIe or from another GPU (0.4% of all 480 routed)
strata serve: KV streaming: 37.57% of 4104 block reads hit VRAM, 10.3 MiB read from RAM
strata batch: slot 2 takes 781 tokens (copied in 179.1 ms)
strata serve: prompt 781 tokens = 0 reused + 781 read in 3603 ms (216.8 tok/s), 1 generated in 86 ms (11.6 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 66.7% (318 hits / 477 lookups); 3 more read by the GPU over PCIe or from another GPU (0.6% of all 480 routed)
strata serve: KV streaming: 79.92% of 3904 block reads hit VRAM, 3.2 MiB read from RAM
strata batch (pipelined, 2 groups of 2): 2537 group-steps, 2537 rows in 85598 ms = 29.6 rows/s (admissions included)
strata serve: prompt 31332 tokens = 0 reused + 31332 read in 31906 ms (982.0 tok/s), 6503 generated in 134804 ms (48.2 tok/s), drafts accepted 3633 of 4571, 3 checkpoints
strata serve: decode expert cache hit rate: 94.7% (3380919 hits / 3570792 lookups); 1368 more read by the GPU over PCIe or from another GPU (0.0% of all 3572160 routed)
strata serve: KV streaming: 99.80% of 15294132 block reads hit VRAM, 123.3 MiB read from RAM
strata serve: suffix drafts: 40 windows, 71 of 105 drafts accepted
strata serve: prompt 34335 tokens = 31327 reused + 3008 read in 6641 ms (453.0 tok/s), 394 generated in 7384 ms (53.4 tok/s), drafts accepted 289 of 299, 4 checkpoints
strata serve: decode expert cache hit rate: 88.7% (172406 hits / 194314 lookups); 566 more read by the GPU over PCIe or from another GPU (0.3% of all 194880 routed)
strata serve: KV streaming: 99.81% of 16136264 block reads hit VRAM, 126.5 MiB read from RAM
strata serve: suffix drafts: 9 windows, 21 of 23 drafts accepted
strata serve: prompt 34901 tokens = 34330 reused + 571 read in 3398 ms (168.1 tok/s), 1080 generated in 22600 ms (47.8 tok/s), drafts accepted 695 of 819, 5 checkpoints
strata serve: decode expert cache hit rate: 91.6% (528855 hits / 577282 lookups); 638 more read by the GPU over PCIe or from another GPU (0.1% of all 577920 routed)
strata serve: KV streaming: 99.83% of 18617476 block reads hit VRAM, 129.1 MiB read from RAM
strata serve: suffix drafts: 58 windows, 225 of 245 drafts accepted
strata serve: prompt 35691 tokens = 34896 reused + 795 read in 3750 ms (212.0 tok/s), 648 generated in 15317 ms (42.3 tok/s), drafts accepted 318 of 430, 6 checkpoints
strata serve: decode expert cache hit rate: 92.7% (339219 hits / 366065 lookups); 175 more read by the GPU over PCIe or from another GPU (0.0% of all 366240 routed)
strata serve: KV streaming: 99.84% of 20192892 block reads hit VRAM, 130.8 MiB read from RAM
strata serve: suffix drafts: 18 windows, 67 of 77 drafts accepted
strata serve: prompt 35909 tokens = 35686 reused + 223 read in 2479 ms (90.0 tok/s), 128 generated in 2293 ms (55.8 tok/s), drafts accepted 91 of 99, 6 checkpoints
strata serve: decode expert cache hit rate: 93.0% (61910 hits / 66589 lookups); 131 more read by the GPU over PCIe or from another GPU (0.2% of all 66720 routed)
strata serve: KV streaming: 99.84% of 20486624 block reads hit VRAM, 130.8 MiB read from RAM
strata serve: suffix drafts: 8 windows, 34 of 36 drafts accepted
strata serve: prompt 29213 tokens = 14932 reused + 14281 read in 16271 ms (877.7 tok/s), 712 generated in 13479 ms (52.8 tok/s), drafts accepted 497 of 573, 2 checkpoints
strata serve: decode expert cache hit rate: 91.6% (347418 hits / 379326 lookups); 354 more read by the GPU over PCIe or from another GPU (0.1% of all 379680 routed)
strata serve: KV streaming: 99.85% of 22119552 block reads hit VRAM, 132.5 MiB read from RAM
strata serve: suffix drafts: 58 windows, 244 of 266 drafts accepted
strata serve: prompt 29921 tokens = 29208 reused + 713 read in 3539 ms (201.5 tok/s), 613 generated in 14217 ms (43.1 tok/s), drafts accepted 331 of 404, 3 checkpoints
strata serve: decode expert cache hit rate: 92.4% (305166 hits / 330126 lookups); 114 more read by the GPU over PCIe or from another GPU (0.0% of all 330240 routed)
strata serve: KV streaming: 99.86% of 23540896 block reads hit VRAM, 134.1 MiB read from RAM
strata serve: suffix drafts: 19 windows, 64 of 79 drafts accepted
strata serve: prompt 30111 tokens = 29916 reused + 195 read in 2402 ms (81.2 tok/s), 356 generated in 7359 ms (48.4 tok/s), drafts accepted 208 of 255, 4 checkpoints
strata serve: decode expert cache hit rate: 92.4% (179982 hits / 194817 lookups); 63 more read by the GPU over PCIe or from another GPU (0.0% of all 194880 routed)
strata serve: KV streaming: 99.86% of 24383040 block reads hit VRAM, 134.5 MiB read from RAM
strata serve: suffix drafts: 21 windows, 85 of 92 drafts accepted
strata serve: prompt 30362 tokens = 30106 reused + 256 read in 2627 ms (97.5 tok/s), 426 generated in 9662 ms (44.1 tok/s), drafts accepted 210 of 273, 5 checkpoints
strata serve: decode expert cache hit rate: 92.3% (217351 hits / 235466 lookups); 214 more read by the GPU over PCIe or from another GPU (0.1% of all 235680 routed)
strata serve: KV streaming: 99.87% of 25399768 block reads hit VRAM, 137.3 MiB read from RAM
strata serve: suffix drafts: 5 windows, 14 of 18 drafts accepted
strata serve: prompt 30629 tokens = 30357 reused + 272 read in 2783 ms (97.7 tok/s), 207 generated in 4342 ms (47.7 tok/s), drafts accepted 147 of 164, 6 checkpoints
strata serve: decode expert cache hit rate: 86.7% (93897 hits / 108344 lookups); 136 more read by the GPU over PCIe or from another GPU (0.1% of all 108480 routed)
strata serve: KV streaming: 99.87% of 25872184 block reads hit VRAM, 137.4 MiB read from RAM
strata serve: suffix drafts: 17 windows, 73 of 80 drafts accepted
strata serve: prompt 30882 tokens = 30624 reused + 258 read in 2593 ms (99.5 tok/s), 184 generated in 4112 ms (44.7 tok/s), drafts accepted 117 of 123, 6 checkpoints
strata serve: decode expert cache hit rate: 88.0% (80593 hits / 91560 lookups); 120 more read by the GPU over PCIe or from another GPU (0.1% of all 91680 routed)
strata serve: KV streaming: 99.87% of 26272720 block reads hit VRAM, 137.7 MiB read from RAM
strata serve: suffix drafts: 6 windows, 26 of 26 drafts accepted
strata serve: prompt 31286 tokens = 0 reused + 31286 read in 31617 ms (989.5 tok/s), 109 generated in 2211 ms (49.3 tok/s), drafts accepted 71 of 74, 3 checkpoints
strata serve: decode expert cache hit rate: 91.4% (49976 hits / 54672 lookups); 48 more read by the GPU over PCIe or from another GPU (0.1% of all 54720 routed)
strata serve: KV streaming: 95.31% of 242372 block reads hit VRAM, 45.8 MiB read from RAM
strata serve: prompt 32020 tokens = 0 reused + 32020 read in 33854 ms (945.8 tok/s), 175 generated in 3471 ms (50.4 tok/s), drafts accepted 112 of 116, 3 checkpoints
strata serve: decode expert cache hit rate: 90.1% (78170 hits / 86805 lookups); 75 more read by the GPU over PCIe or from another GPU (0.1% of all 86880 routed)
strata serve: KV streaming: 96.83% of 379984 block reads hit VRAM, 48.6 MiB read from RAM
strata serve: suffix drafts: 1 windows, 1 of 1 drafts accepted
strata serve: prompt 32219 tokens = 14932 reused + 17287 read in 19978 ms (865.3 tok/s), 159 generated in 3299 ms (48.2 tok/s), drafts accepted 104 of 110, 3 checkpoints
strata serve: decode expert cache hit rate: 89.4% (72408 hits / 81019 lookups); 101 more read by the GPU over PCIe or from another GPU (0.1% of all 81120 routed)
strata serve: KV streaming: 97.77% of 735324 block reads hit VRAM, 65.9 MiB read from RAM
strata serve: suffix drafts: 12 windows, 39 of 39 drafts accepted
strata serve: prompt 32887 tokens = 14932 reused + 17955 read in 20293 ms (884.8 tok/s), 259 generated in 5751 ms (45.0 tok/s), drafts accepted 147 of 183, 3 checkpoints
strata serve: decode expert cache hit rate: 93.2% (132317 hits / 141993 lookups); 87 more read by the GPU over PCIe or from another GPU (0.1% of all 142080 routed)
strata serve: KV streaming: 98.44% of 1351536 block reads hit VRAM, 85.0 MiB read from RAM
strata serve: suffix drafts: 2 windows, 2 of 4 drafts accepted
strata serve: prompt 31547 tokens = 14932 reused + 16615 read in 18919 ms (878.2 tok/s), 487 generated in 10820 ms (45.0 tok/s), drafts accepted 257 of 329, 3 checkpoints
strata serve: decode expert cache hit rate: 92.3% (247627 hits / 268184 lookups); 136 more read by the GPU over PCIe or from another GPU (0.1% of all 268320 routed)
strata serve: KV streaming: 99.03% of 2507932 block reads hit VRAM, 98.4 MiB read from RAM
strata serve: suffix drafts: 9 windows, 13 of 21 drafts accepted
strata serve: prompt 31904 tokens = 14932 reused + 16972 read in 19447 ms (872.7 tok/s), 179 generated in 4714 ms (38.0 tok/s), drafts accepted 77 of 105, 3 checkpoints
strata serve: decode expert cache hit rate: 92.1% (92804 hits / 100771 lookups); 29 more read by the GPU over PCIe or from another GPU (0.0% of all 100800 routed)
strata serve: KV streaming: 99.16% of 2947488 block reads hit VRAM, 99.7 MiB read from RAM
strata serve: suffix drafts: 2 windows, 6 of 6 drafts accepted
strata serve: prompt 33066 tokens = 14932 reused + 18134 read in 20286 ms (893.9 tok/s), 1079 generated in 24938 ms (43.3 tok/s), drafts accepted 471 of 647, 3 checkpoints
strata serve: decode expert cache hit rate: 94.6% (569546 hits / 602146 lookups); 254 more read by the GPU over PCIe or from another GPU (0.0% of all 602400 routed)
strata serve: KV streaming: 99.50% of 5533504 block reads hit VRAM, 111.2 MiB read from RAM
strata serve: suffix drafts: 8 windows, 4 of 15 drafts accepted
strata serve: prompt 33334 tokens = 14932 reused + 18402 read in 20755 ms (886.6 tok/s), 467 generated in 10480 ms (44.6 tok/s), drafts accepted 255 of 358, 3 checkpoints
strata serve: decode expert cache hit rate: 94.9% (259418 hits / 273415 lookups); 185 more read by the GPU over PCIe or from another GPU (0.1% of all 273600 routed)
strata serve: KV streaming: 99.58% of 6712520 block reads hit VRAM, 112.9 MiB read from RAM
strata serve: prompt 3743 tokens = 0 reused + 3743 read in 6542 ms (572.2 tok/s), 141 generated in 3722 ms (37.9 tok/s), drafts accepted 69 of 108, 1 checkpoints
strata serve: decode expert cache hit rate: 91.4% (80255 hits / 87786 lookups); 54 more read by the GPU over PCIe or from another GPU (0.1% of all 87840 routed)
strata serve: KV streaming: 99.02% of 384092 block reads hit VRAM, 15.2 MiB read from RAM
strata serve: suffix drafts: 1 windows, 1 of 3 drafts accepted
strata serve: prompt 309 tokens = 0 reused + 309 read in 3094 ms (99.9 tok/s), 1 generated in 27 ms (36.5 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 90.6% (435 hits / 480 lookups)
strata serve: KV streaming: 79.79% of 1544 block reads hit VRAM, 1.3 MiB read from RAM
strata batch: slot 0 takes 310 tokens (copied in 155.9 ms)
strata serve: prompt 310 tokens = 309 reused + 1 read in 0 ms (3528.6 tok/s), 1 generated in 24 ms (41.7 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 91.7% (440 hits / 480 lookups)
strata serve: KV streaming: 83.19% of 1856 block reads hit VRAM, 1.3 MiB read from RAM
strata batch: slot 2 takes 19129 tokens (copied in 414.9 ms)
strata serve: prompt 19129 tokens = 0 reused + 19129 read in 28903 ms (661.8 tok/s), 1 generated in 27 ms (36.7 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: decode expert cache hit rate: 96.9% (465 hits / 480 lookups)
strata serve: KV streaming: 64.93% of 10272 block reads hit VRAM, 14.5 MiB read from RAM
strata batch (pipelined, 2 groups of 2): 5 group-steps, 5 rows in 29486 ms = 0.2 rows/s (admissions included)
strata batch: slot 2 gave back 19132 tokens of this conversation (all it holds) in 365.7 ms
strata serve: prompt 19133 tokens = 19132 reused + 1 read in 368 ms (2.7 tok/s), 158 generated in 3903 ms (40.5 tok/s), drafts accepted 100 of 124, 0 checkpoints
strata serve: decode expert cache hit rate: 89.4% (78421 hits / 87747 lookups); 93 more read by the GPU over PCIe or from another GPU (0.1% of all 87840 routed)
strata serve: KV streaming: 97.23% of 375884 block reads hit VRAM, 41.9 MiB read from RAM
strata serve: suffix drafts: 3 windows, 15 of 15 drafts accepted
strata serve: prompt 25089 tokens = 0 reused + 25089 read in 37530 ms (668.5 tok/s), 762 generated in 19694 ms (38.7 tok/s), drafts accepted 400 of 520, 3 checkpoints
strata serve: decode expert cache hit rate: 94.8% (401846 hits / 423762 lookups); 78 more read by the GPU over PCIe or from another GPU (0.0% of all 423840 routed)
strata serve: KV streaming: 99.05% of 1821904 block reads hit VRAM, 69.5 MiB read from RAM
strata serve: suffix drafts: 7 windows, 32 of 35 drafts accepted
strata serve: prompt 28196 tokens = 25084 reused + 3112 read in 7730 ms (402.6 tok/s), 945 generated in 24387 ms (38.8 tok/s), drafts accepted 506 of 673, 4 checkpoints
strata serve: decode expert cache hit rate: 95.7% (511144 hits / 534160 lookups); 80 more read by the GPU over PCIe or from another GPU (0.0% of all 534240 routed)
strata serve: KV streaming: 99.49% of 4116232 block reads hit VRAM, 84.2 MiB read from RAM
strata serve: suffix drafts: 13 windows, 48 of 52 drafts accepted
strata serve: prompt 33222 tokens = 28191 reused + 5031 read in 10188 ms (493.8 tok/s), 731 generated in 18537 ms (39.4 tok/s), drafts accepted 379 of 490, 5 checkpoints
strata serve: decode expert cache hit rate: 95.8% (387619 hits / 404605 lookups); 35 more read by the GPU over PCIe or from another GPU (0.0% of all 404640 routed)
strata serve: KV streaming: 99.56% of 5855976 block reads hit VRAM, 103.5 MiB read from RAM
strata serve: suffix drafts: 13 windows, 36 of 41 drafts accepted
strata serve: prompt 37284 tokens = 33217 reused + 4067 read in 9203 ms (441.9 tok/s), 5691 generated in 125513 ms (45.3 tok/s), drafts accepted 3804 of 4401, 6 checkpoints
strata serve: decode expert cache hit rate: 96.0% (2898153 hits / 3018995 lookups); 685 more read by the GPU over PCIe or from another GPU (0.0% of all 3019680 routed)
strata serve: KV streaming: 99.80% of 18785892 block reads hit VRAM, 150.9 MiB read from RAM
strata serve: suffix drafts: 48 windows, 151 of 183 drafts accepted
strata serve: prompt 249 tokens = 0 reused + 249 read in 3807 ms (65.4 tok/s), 1 generated in 51 ms (19.5 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 70.1% (336 hits / 479 lookups); 1 more read by the GPU over PCIe or from another GPU (0.2% of all 480 routed)
strata serve: KV streaming: 79.74% of 1244 block reads hit VRAM, 1.0 MiB read from RAM
strata batch: slot 0 takes 250 tokens (copied in 161.1 ms)
strata serve: prompt 250 tokens = 249 reused + 1 read in 0 ms (2998.5 tok/s), 1 generated in 49 ms (20.6 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata serve: decode expert cache hit rate: 70.6% (339 hits / 480 lookups)
strata serve: KV streaming: 83.16% of 1496 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: no progress for 60 s during a request (idle -1) - stopping the engine so the server starts it again (issue #29)
strata serve: stall report (engine 0.1.43): stage "idle -1" for 60 s; 0 layers served since the last finished step (0 = stopped, more = slow)
  expert pool: epoch 1068244, batch epoch 1068244: 54 of 54 jobs claimed, 54 done; 17 of 17 workers parked, 17 sleeping; mode 0
  expert pool threads: w0=sleeping w1=sleeping w2=sleeping w3=sleeping w4=sleeping w5=sleeping w6=sleeping w7=sleeping w8=sleeping w9=sleeping w10=sleeping w11=sleeping w12=sleeping w13=sleeping w14=sleeping w15=sleeping w16=sleeping; host idle for 60810 ms
  verify window (last window, not the current stage): 1 tokens at position 249, host at layer step 18; the GPU rang 18; flags: served 18, plan (A) 18, copies (B) 18
  memory: 21074 MiB resident, 109200 MiB committed, 41776 MiB RAM available; 13384988 page faults so far
  2 s later:
  expert pool: epoch 1068244, batch epoch 1068244: 54 of 54 jobs claimed, 54 done; 17 of 17 workers parked, 17 sleeping; mode 0
  expert pool threads: w0=sleeping w1=sleeping w2=sleeping w3=sleeping w4=sleeping w5=sleeping w6=sleeping w7=sleeping w8=sleeping w9=sleeping w10=sleeping w11=sleeping w12=sleeping w13=sleeping w14=sleeping w15=sleeping w16=sleeping; host idle for 62817 ms
  verify window (last window, not the current stage): 1 tokens at position 249, host at layer step 18; the GPU rang 18; flags: served 18, plan (A) 18, copies (B) 18
  memory: 20864 MiB resident, 109200 MiB committed, 41985 MiB RAM available; 13384988 page faults so far
  wrote the thread stacks to C:\Strata-0.1.43\strata-stall-28072.dmp (attach it to the issue)
strata: released the verify window's GPU waits (#267): the GPU finished in 17 ms
strata: released the verify window's GPU waits (#267): the GPU finished in 0 ms
[strata] 2026-10-13 07:18:43 engine started: C:\Strata-0.1.43\engine\strata.exe --serve --pack C:\Strata-data\packs\iq3_s --native C:\Strata-data\models\IQ3_S\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf C:\Strata-data\models\IQ3_S\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile C:\Strata-0.1.43\data\expert-profile.bin --expert-cache auto --prefill auto --spec 4 --mtp C:\Strata-data\mtp\rt --max-context 262144 --kv int8 --kv-resident 32768 --vision --vram-reserve-mib 700 --control-vector-scaled C:\Strata-0.1.43\data\experimental-speed-projection\Qwen3.8-Flash-Next-experimental-speed-projection.gguf:1.0 --control-vector-layer-range 4 44 --cvec-mode project --cvec-dir per-layer --pcie-frac 0.55 --spec-min-p 0.70 --pool-workers 17 --batch 4 --layer-split auto
strata generate: memory guard: limit 511.9 GiB (source: meminfo), current 403.7 GiB, reclaimable cache credited 0.0 GiB; available 108.2 GiB (MemAvailable 108.2 GiB), floor 2560 MiB
strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: C:\Strata-data\packs\iq3_s experts (largest blob 2.66 MB), token embedding IQ4_XS in mapped host memory (322 MiB, 2.2 s)
strata generate: 1466 MiB of weights loaded from C:\Strata-data\packs\iq3_s in 1.0 s (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 2018.88 MiB of weights, in 10.9 s
strata generate: control vector mode = project, dir = per-layer, layers 4..44 (41 steered)
strata generate: PLE direct read probe: 25908 rows/s (fine)
strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off)
strata generate: layer split: CUDA1 holds its weights; 17.15 GiB free (its session follows the split search)
strata generate: profile C:\Strata-0.1.43\data\expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: layer split auto: CUDA0 68 SMs at 1.54 GHz -> 0.69 ms per layer, 16.47 GiB free before its session carve
strata generate: layer split auto: CUDA1 68 SMs at 1.54 GHz -> 0.69 ms per layer, 15.40 GiB free before its session carve
strata generate: layer split auto: CUDA0 link 5.9 GB/s -> miss cost x3.38
strata generate: layer split auto: CUDA1 link 11.0 GB/s -> miss cost x1.82
strata generate: layer split auto: K=18 - predicted 37.7 ms per decode window; the caches hold 16487 of 24576 profiled pairs (~98.7% of the routed mass), best of 46 placements
strata generate: layer split auto: NEAR TIE - K=21 is predicted 38.0 ms per decode window, within 0.6% of K=18 (37.7 ms); the cards differ, so the prediction may not rank them (it prices decode, not the prompt). To try the other: --layer-split 21
strata generate: KV streaming: 32768 of 262144 cells per QSA layer in VRAM, the K/V in 1.03 GiB of pinned RAM
strata generate: PLE on, table 320001536 rows of C:\Strata-data\models\IQ3_S\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf
strata generate: layer split: CUDA1 holds its weights, session [18, 48) and the head; 16.02 GiB free
strata generate: --batch-groups auto: 2 groups of 2 slots
strata generate: --batch: 4 slot sessions on CUDA0 (0.37 GiB each); 15.30 GiB free
strata generate: --batch: 4 slot sessions on CUDA1 (0.65 GiB each); 13.42 GiB free
strata generate: --batch 4: the slot sessions take 1.48 GiB of VRAM on CUDA0 that the expert cache would otherwise hold
strata mtp: draft layer loaded, 835 MiB of VRAM (experts 675, dense 111), files read in 0.76 s (1027 MiB/s)
strata generate: GPU 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5
strata generate: multi-GPU under WDDM: at most 8 GiB of the expert arena is pinned (STRATA_ARENA_PIN_GIB changes it)
strata generate: expert arena read unbuffered (0 of 16 probe reads from the file cache; 45.0 GiB available, 51.1 GiB of files)
strata generate: expert arena: large pages are resident without a lock; cudaHostRegister limited to 8 GiB by the engine (multi-GPU under WDDM, or remote experts); 8 slices pinned (7 GiB); large pages (2097152 B)
strata generate: loaded 46.84 GiB at 0.41 GiB/s
strata generate: CPU pool tasks/phase: 54 (automatic), participating threads: 18
strata generate: 17 pool workers on logical processors 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17, the host thread on 0 (draining too) (--host-core first)
strata aux cpus: not available on this system
strata generate: expert cache auto: 15.30 GiB free, 700 MiB reserved (+0 MiB for the draft head) -> 5894 slots
strata generate: only 0 MiB free once the slots are written (reserve 700 MiB); shrinking the expert cache
strata generate: only 0 MiB free once the slots are written (reserve 700 MiB); shrinking the expert cache
strata generate: only 0 MiB free once the slots are written (reserve 700 MiB); shrinking the expert cache
strata generate: expert cache 5308 slots, 9.46 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
                 so a reply can differ slightly from a run without the cache (same quality:
                 bench/results/2026-09-27-cache-parity).
                 PROFILE, ranked by routing frequency, no eviction.
strata generate: pre-filled 5308 of 5308 slots from the profile in 2.3 s (4382 MB/s); slot 0 verified
strata generate: layer split, CUDA1: 12.60 GiB free of 22.00, room for experts 11.82 GiB
strata generate: layer split: CUDA1 runs layers 18-47, expert cache 5993 slots (11.82 GiB), 5993 of its 15360 profiled pairs; slot 0 verified
strata generate: layer split: CUDA0 runs layers 0-17
strata generate: layer split auto: L2 probe: the split is startable on the live free figures
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 17 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.43)
strata generate: token graph hit path: 11301 resident experts, decided on the device
strata serve: Windows commit: 17.2 GiB left of a 527.9 GiB limit (RAM + page file)
strata serve: prompt chunk auto: 8192 tokens, a 96-slot ring
strata serve:   CUDA0 keeps its own prompt buffers (3.66 GiB of 5.84 GiB free): no loan
strata serve:   CUDA1 prompt path borrows 1865 of its 5993 slots (3.66 GiB)
strata hc: CUDA1: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata verify: window up to 6 tokens, 75.8 MiB of device buffers
strata serve: layer split: layers 0-17 (CUDA0), 18-47 (CUDA1), one hand-off per window
strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata verify: window up to 6 tokens, 75.8 MiB of device buffers
strata mtp: draft head over 106299 tokens (212.9 MiB)
strata verify: batch windows of up to 4 sequences (layers [0, 18))
strata verify: batch windows of up to 4 sequences (layers [18, 48))
strata serve: 0 MiB of VRAM free with everything loaded - LOW: requests may stall; add --vram-reserve-mib 1212 to the config's args (or lower --max-context)
strata verify: capturing the 4-token window (0 MiB of VRAM free)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: capturing the 4-token window (158 MiB of VRAM free)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: capturing the 1-token window (0 MiB of VRAM free)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: capturing the 1-token window (135 MiB of VRAM free)
strata verify: captured the 1-token window (upload no error, sync no error)
strata serve: prompt 18518 tokens = 0 reused + 18518 read in 26917 ms (688.0 tok/s), 1 generated in 77 ms (12.9 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: decode expert cache hit rate: 81.7% (388 hits / 475 lookups); 5 more read by the GPU over PCIe or from another GPU (1.0% of all 480 routed)
strata serve: KV streaming: 63.60% of 10272 block reads hit VRAM, 15.1 MiB read from RAM
strata batch: slot 0 takes 18519 tokens (copied in 436.9 ms)
strata serve: prompt 18519 tokens = 18518 reused + 1 read in 2 ms (505.8 tok/s), 1 generated in 44 ms (22.7 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: decode expert cache hit rate: 86.0% (412 hits / 479 lookups); 1 more read by the GPU over PCIe or from another GPU (0.2% of all 480 routed)
strata serve: KV streaming: 68.06% of 12324 block reads hit VRAM, 15.9 MiB read from RAM
strata verify: captured the batch window over slots 0
strata verify: captured the batch window over slots 0
strata serve: no progress for 60 s during a request (idle -1) - stopping the engine so the server starts it again (issue #29)
strata serve: stall report (engine 0.1.43): stage "idle -1" for 60 s; 0 layers served since the last finished step (0 = stopped, more = slow)
  expert pool: epoch 280, batch epoch 280: 54 of 54 jobs claimed, 54 done; 17 of 17 workers parked, 17 sleeping; mode 0
  expert pool threads: w0=sleeping w1=sleeping w2=sleeping w3=sleeping w4=sleeping w5=sleeping w6=sleeping w7=sleeping w8=sleeping w9=sleeping w10=sleeping w11=sleeping w12=sleeping w13=sleeping w14=sleeping w15=sleeping w16=sleeping; host idle for 60747 ms
  verify window (last window, not the current stage): 1 tokens at position 18518, host at layer step 18; the GPU rang 18; flags: served 18, plan (A) 18, copies (B) 18
  memory: 20227 MiB resident, 109123 MiB committed, 42525 MiB RAM available; 7186093 page faults so far
  2 s later:
  expert pool: epoch 280, batch epoch 280: 54 of 54 jobs claimed, 54 done; 17 of 17 workers parked, 17 sleeping; mode 0
  expert pool threads: w0=sleeping w1=sleeping w2=sleeping w3=sleeping w4=sleeping w5=sleeping w6=sleeping w7=sleeping w8=sleeping w9=sleeping w10=sleeping w11=sleeping w12=sleeping w13=sleeping w14=sleeping w15=sleeping w16=sleeping; host idle for 62748 ms
  verify window (last window, not the current stage): 1 tokens at position 18518, host at layer step 18; the GPU rang 18; flags: served 18, plan (A) 18, copies (B) 18
  memory: 20227 MiB resident, 109123 MiB committed, 42402 MiB RAM available; 7186093 page faults so far
  wrote the thread stacks to C:\Strata-0.1.43\strata-stall-21128.dmp (attach it to the issue)
strata: released the verify window's GPU waits (#267): the GPU finished in 11 ms
strata: released the verify window's GPU waits (#267): the GPU finished in 0 ms
load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842