Repository navigation
--gpus with mixed architectures fails: engine not rebuilt when adding a GPU with a different compute capability #128
Description
Bug
When starting Strata with --gpus that includes a GPU whose compute capability differs from what the engine was originally built for, the engine is not recompiled. It crashes with a CUDA error on the first kernel launch on the unsupported card.
Reproduction
- Install and run Strata with 2 GPUs of the same generation (e.g. 2× RTX 5060 Ti, both sm_120). The engine is compiled with CMAKE_CUDA_ARCHITECTURES=120.
- Add a third GPU of a different generation (e.g. RTX 3060, sm_86).
- Run:
./setup.sh --gpus 0,1,2 --calibrate
Result: The engine starts, splits layers across all 3 GPUs, then crashes on the first kernel launch on the sm_86 card:
strata generate: layer split across 3 GPUs: CUDA0, then CUDA1,2 (split auto)
...
strata generate: layer split: CUDA1 holds its weights, session; 11.42 GiB free
kv_stream: reset: no kernel image is available for execution on the device
The same happens without --calibrate: any start() path that resolves to a multi-GPU list whose archs are not a subset of the engine's built archs.
Root cause
setup.py → start() (line ~1192) and the --calibrate shortcut (line ~1338) never verify that the installed engine binary was compiled for all GPU architectures in the --gpus list. The engine_runs_on() helper exists (line 365) but is only called from offer_together() (line 420), not from the --gpus/start() path.
The build-time logic in build_engine() (line 865) correctly unions archs from gpu["archs"], but that function is only reached during the full --setup flow (step 1 → step 3), not when starting an already-installed model with a different GPU set.
Expected behaviour
When start() (or the calibrate path) resolves to a GPU list whose architectures are not all covered by the engine's BUILD.json "archs" field, it should recompile the engine for the full arch set (reusing build_engine()) before launching, or at minimum warn and refuse to start.
Suggested fix
In start(), after the GPU list is resolved (after line ~1217), add:
use = gpu if gpu is not None else cfg.get("gpu")
if isinstance(use, list) and len(use) > 1:
byid = {g["index"]: g for g in found}
needed_archs = sorted({int(byid[i]["arch"]) for i in use if i in byid})
ea = engine_archs()
if ea is not None and ea[0]:
missing = [a for a in needed_archs
if a not in ea[0] and not (ea[1] and a > max(ea[0]))]
if missing:
say(f" The engine has no code for {', '.join('sm_' + str(a) for a in missing)}: recompiling ...")
build_engine({"archs": needed_archs}, "none", yes, get_llama_cpp())
This reuses the existing build_engine() which handles the BUILD.json stamp, source-hash check, and all bookkeeping. The same guard should be added to the --calibrate shortcut at line 1348 (before start() is called) or, better, placed inside start() itself so both paths are covered.
Environment
- OS: Linux
- GPUs: 2× RTX 5060 Ti (sm120) + 1× RTX 3060 (sm86)
- Strata engine: 0.1.22
- Command: ./setup.sh --gpus 0,1,2 --calibrate
model Log
strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)
strata generate: PCIe probe: 26.5 GB/s host->device -> pcie_frac 0.55 (default 0.55)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: /home/mike/Programs/Strata-data/packs/swift-iq3_xxs experts (largest blob 2.33 MB), token embedding type 21 in mapped host memory (260 MiB)
strata generate: 1466 MiB of weights loaded from /home/mike/Programs/Strata-data/packs/swift-iq3_xxs (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 1781.11 MiB of weights
strata generate: KV streaming: 32768 of 131072 cells per QSA layer in VRAM, the K/V in 0.84 GiB of pinned RAM
strata generate: PLE on, table 320001536 rows of /home/mike/Programs/Strata-data/models/swift-IQ3_XXS/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf
strata generate: layer split: CUDA1 PCIe probe 6.7 GB/s -> pcie_frac 0.14
strata generate: layer split: CUDA1 holds its weights, session and head; 11.01 GiB free
strata mtp: draft layer loaded, 820 MiB of VRAM (experts 675, dense 111)
strata generate: expert arena: mlock failed (raise ulimit -l); cudaHostRegister limited to 8 GiB for CUDA1; 10 slices pinned (7 GiB); MAP_HUGETLB unavailable (no hugetlb pool configured?); using 4 KB pages
strata generate: loaded 39.97 GiB at 4.53 GiB/s
strata generate: profile /home/mike/Programs/Strata/data/expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: layer split auto: CUDA0 36 SMs at 2.60 GHz -> 0.77 ms per layer, 8.72 GiB for experts
strata generate: layer split auto: CUDA1 36 SMs at 2.60 GHz -> 0.77 ms per layer, 7.03 GiB for experts
strata generate: layer split auto: K=26 - predicted 42.4 ms per decode window; the caches hold 9808 of 24576 profiled pairs (~97.3% of the routed mass)
strata generate: expert cache auto: 10.88 GiB free, 700 MiB reserved -> 4017 slots
strata generate: expert cache 5990 slots, 8.71 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
so a reply can differ slightly from a run without the cache (same quality:
bench/results/2026-09-27-cache-parity).
PROFILE, ranked by routing frequency, no eviction.
strata generate: pre-filled 5990 of 5990 slots from the profile; slot 0 verified
strata generate: layer split: CUDA1 runs layers 26-47, expert cache 3817 slots (7.02 GiB), 3817 of its 11264 profiled pairs; slot 0 verified
strata generate: layer split: CUDA0 runs layers 0-25
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 13 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.22)
strata generate: token graph hit path: 9807 resident experts, decided on the device
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
strata verify: window up to 6 tokens, 69.6 MiB of device buffers
strata serve: layer split: layers 0-25 (CUDA0), 26-47 (CUDA1), one hand-off per window
strata verify: window up to 6 tokens, 69.6 MiB of device buffers
strata mtp: draft head over 40525 tokens (68.0 MiB)
strata serve: 341 MiB of VRAM free with everything loaded
strata verify: captured the 6-token window (upload no error, sync no error)
strata verify: captured the 6-token window (upload no error, sync no error)
strata verify: captured the 3-token window (upload no error, sync no error)
strata verify: captured the 3-token window (upload no error, sync no error)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: captured the 2-token window (upload no error, sync no error)
strata verify: captured the 2-token window (upload no error, sync no error)
strata serve: prompt 34 tokens = 0 reused + 34 read in 493 ms (69.0 tok/s), 128 generated in 1926 ms (66.5 tok/s), drafts accepted 90 of 101, 1 checkpoints
strata serve: decode expert cache hit rate: 66.4% (43142 hits / 64993 lookups)
strata serve: KV streaming: 98.85% of 21318 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 2 windows, 3 of 4 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 401 ms (67.3 tok/s), 128 generated in 2400 ms (53.3 tok/s), drafts accepted 69 of 126, 1 checkpoints
strata serve: decode expert cache hit rate: 83.5% (73202 hits / 87662 lookups)
strata serve: KV streaming: 99.09% of 25662 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 362 ms (60.8 tok/s), 128 generated in 2283 ms (56.1 tok/s), drafts accepted 75 of 124, 1 checkpoints
strata serve: decode expert cache hit rate: 77.7% (65286 hits / 83976 lookups)
strata serve: KV streaming: 99.04% of 23874 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 414 ms (82.1 tok/s), 128 generated in 1773 ms (72.2 tok/s), drafts accepted 89 of 105, 1 checkpoints
strata serve: decode expert cache hit rate: 77.6% (52778 hits / 67986 lookups)
strata serve: KV streaming: 98.90% of 22278 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 1 windows, 3 of 3 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 356 ms (75.9 tok/s), 128 generated in 2355 ms (54.4 tok/s), drafts accepted 64 of 138, 1 checkpoints
strata serve: decode expert cache hit rate: 92.6% (89500 hits / 96704 lookups)
strata serve: KV streaming: 99.16% of 28572 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 311 ms (70.8 tok/s), 128 generated in 2067 ms (61.9 tok/s), drafts accepted 75 of 118, 1 checkpoints
strata serve: decode expert cache hit rate: 88.3% (71894 hits / 81451 lookups)
strata serve: KV streaming: 99.02% of 23316 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 407 ms (83.5 tok/s), 128 generated in 1865 ms (68.6 tok/s), drafts accepted 87 of 115, 1 checkpoints
strata serve: decode expert cache hit rate: 74.9% (56468 hits / 75360 lookups)
strata serve: KV streaming: 98.97% of 23988 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 1 windows, 3 of 5 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 351 ms (77.0 tok/s), 128 generated in 2330 ms (54.9 tok/s), drafts accepted 64 of 139, 1 checkpoints
strata serve: decode expert cache hit rate: 92.0% (90089 hits / 97920 lookups)
strata serve: KV streaming: 99.16% of 28680 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 312 ms (70.4 tok/s), 128 generated in 2059 ms (62.2 tok/s), drafts accepted 69 of 113, 1 checkpoints
strata serve: decode expert cache hit rate: 87.9% (72545 hits / 82560 lookups)
strata serve: KV streaming: 99.01% of 22926 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 418 ms (81.3 tok/s), 128 generated in 1824 ms (70.2 tok/s), drafts accepted 87 of 111, 1 checkpoints
strata serve: decode expert cache hit rate: 74.3% (54213 hits / 72957 lookups)
strata serve: KV streaming: 98.95% of 23424 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 2 windows, 4 of 5 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 356 ms (75.9 tok/s), 128 generated in 2251 ms (56.9 tok/s), drafts accepted 66 of 130, 1 checkpoints
strata serve: decode expert cache hit rate: 92.0% (85160 hits / 92574 lookups)
strata serve: KV streaming: 99.14% of 28020 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 318 ms (69.2 tok/s), 128 generated in 2004 ms (63.9 tok/s), drafts accepted 72 of 113, 1 checkpoints
strata serve: decode expert cache hit rate: 88.6% (71706 hits / 80968 lookups)
strata serve: KV streaming: 98.96% of 21870 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 423 ms (80.4 tok/s), 128 generated in 1998 ms (64.1 tok/s), drafts accepted 88 of 113, 1 checkpoints
strata serve: decode expert cache hit rate: 73.2% (53654 hits / 73258 lookups)
strata serve: KV streaming: 98.95% of 23484 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 1 windows, 2 of 2 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 363 ms (74.4 tok/s), 128 generated in 2381 ms (53.8 tok/s), drafts accepted 68 of 141, 1 checkpoints
strata serve: decode expert cache hit rate: 91.5% (88752 hits / 97028 lookups)
strata serve: KV streaming: 99.19% of 29484 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 335 ms (65.8 tok/s), 128 generated in 2132 ms (60.0 tok/s), drafts accepted 73 of 123, 1 checkpoints
strata serve: decode expert cache hit rate: 89.4% (76786 hits / 85874 lookups)
strata serve: KV streaming: 99.08% of 24732 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 423 ms (80.4 tok/s), 128 generated in 1945 ms (65.8 tok/s), drafts accepted 89 of 110, 1 checkpoints
strata serve: decode expert cache hit rate: 74.4% (52134 hits / 70056 lookups)
strata serve: KV streaming: 98.93% of 22914 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 1 windows, 2 of 2 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 367 ms (73.6 tok/s), 128 generated in 2392 ms (53.5 tok/s), drafts accepted 61 of 135, 1 checkpoints
strata serve: decode expert cache hit rate: 92.3% (88861 hits / 96248 lookups)
strata serve: KV streaming: 99.18% of 28446 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 321 ms (68.5 tok/s), 128 generated in 2114 ms (60.5 tok/s), drafts accepted 73 of 122, 1 checkpoints
strata serve: decode expert cache hit rate: 90.3% (76769 hits / 85009 lookups)
strata serve: KV streaming: 99.06% of 24264 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 427 ms (79.6 tok/s), 128 generated in 1871 ms (68.4 tok/s), drafts accepted 90 of 107, 1 checkpoints
strata serve: decode expert cache hit rate: 75.8% (51045 hits / 67363 lookups)
strata serve: KV streaming: 98.90% of 22392 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 1 windows, 2 of 2 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 366 ms (73.9 tok/s), 128 generated in 2349 ms (54.5 tok/s), drafts accepted 63 of 135, 1 checkpoints
strata serve: decode expert cache hit rate: 92.9% (88171 hits / 94959 lookups)
strata serve: KV streaming: 99.17% of 28098 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 319 ms (68.9 tok/s), 128 generated in 2123 ms (60.3 tok/s), drafts accepted 73 of 118, 1 checkpoints
strata serve: decode expert cache hit rate: 91.0% (75253 hits / 82691 lookups)
strata serve: KV streaming: 99.03% of 23598 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 412 ms (82.6 tok/s), 128 generated in 1853 ms (69.1 tok/s), drafts accepted 89 of 117, 1 checkpoints
strata serve: decode expert cache hit rate: 73.1% (54769 hits / 74878 lookups)
strata serve: KV streaming: 98.96% of 23694 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 1 windows, 3 of 3 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 359 ms (75.1 tok/s), 128 generated in 2414 ms (53.0 tok/s), drafts accepted 67 of 155, 1 checkpoints
strata serve: decode expert cache hit rate: 91.9% (95194 hits / 103619 lookups)
strata serve: KV streaming: 99.25% of 31278 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 310 ms (70.9 tok/s), 128 generated in 2150 ms (59.5 tok/s), drafts accepted 75 of 138, 1 checkpoints
strata serve: decode expert cache hit rate: 89.4% (82704 hits / 92495 lookups)
strata serve: KV streaming: 99.13% of 26208 block reads hit VRAM, 0.9 MiB read from RAM
strata verify: captured the 5-token window (upload no error, sync no error)
strata verify: captured the 5-token window (upload no error, sync no error)
strata serve: prompt 34 tokens = 0 reused + 34 read in 418 ms (81.3 tok/s), 128 generated in 1775 ms (72.1 tok/s), drafts accepted 90 of 106, 1 checkpoints
strata serve: decode expert cache hit rate: 72.0% (49725 hits / 69107 lookups)
strata serve: KV streaming: 98.89% of 22260 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 1 windows, 3 of 4 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 354 ms (76.2 tok/s), 128 generated in 2201 ms (58.2 tok/s), drafts accepted 70 of 128, 1 checkpoints
strata serve: decode expert cache hit rate: 91.0% (82007 hits / 90153 lookups)
strata serve: KV streaming: 99.12% of 27420 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 318 ms (69.3 tok/s), 128 generated in 2025 ms (63.2 tok/s), drafts accepted 74 of 120, 1 checkpoints
strata serve: decode expert cache hit rate: 89.7% (75668 hits / 84359 lookups)
strata serve: KV streaming: 99.04% of 23856 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 423 ms (80.3 tok/s), 128 generated in 1731 ms (74.0 tok/s), drafts accepted 86 of 93, 1 checkpoints
strata serve: decode expert cache hit rate: 74.1% (48079 hits / 64870 lookups)
strata serve: KV streaming: 98.84% of 21186 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 1 windows, 3 of 3 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 359 ms (75.2 tok/s), 128 generated in 2116 ms (60.5 tok/s), drafts accepted 61 of 91, 1 checkpoints
strata serve: decode expert cache hit rate: 91.3% (70500 hits / 77226 lookups)
strata serve: KV streaming: 98.94% of 22560 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 307 ms (71.6 tok/s), 128 generated in 1925 ms (66.5 tok/s), drafts accepted 70 of 91, 1 checkpoints
strata serve: decode expert cache hit rate: 89.4% (64257 hits / 71899 lookups)
strata serve: KV streaming: 98.87% of 20256 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 426 ms (79.9 tok/s), 128 generated in 1792 ms (71.4 tok/s), drafts accepted 90 of 107, 1 checkpoints
strata serve: decode expert cache hit rate: 76.4% (52264 hits / 68372 lookups)
strata serve: KV streaming: 98.88% of 22032 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 1 windows, 2 of 2 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 357 ms (75.6 tok/s), 128 generated in 2321 ms (55.1 tok/s), drafts accepted 63 of 132, 1 checkpoints
strata serve: decode expert cache hit rate: 92.1% (86405 hits / 93783 lookups)
strata serve: KV streaming: 99.14% of 27348 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 312 ms (70.4 tok/s), 128 generated in 2144 ms (59.7 tok/s), drafts accepted 73 of 126, 1 checkpoints
strata serve: decode expert cache hit rate: 90.4% (78511 hits / 86893 lookups)
strata serve: KV streaming: 99.08% of 24888 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 410 ms (82.8 tok/s), 128 generated in 1766 ms (72.5 tok/s), drafts accepted 86 of 98, 1 checkpoints
strata serve: decode expert cache hit rate: 74.1% (49821 hits / 67254 lookups)
strata serve: KV streaming: 98.88% of 21954 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 1 windows, 3 of 3 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 355 ms (76.1 tok/s), 128 generated in 2162 ms (59.2 tok/s), drafts accepted 58 of 96, 1 checkpoints
strata serve: decode expert cache hit rate: 91.4% (73226 hits / 80105 lookups)
strata serve: KV streaming: 99.01% of 23562 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 305 ms (72.1 tok/s), 128 generated in 2029 ms (63.1 tok/s), drafts accepted 70 of 106, 1 checkpoints
strata serve: decode expert cache hit rate: 90.2% (71819 hits / 79585 lookups)
strata serve: KV streaming: 99.00% of 22800 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 416 ms (81.8 tok/s), 128 generated in 1837 ms (69.7 tok/s), drafts accepted 89 of 106, 1 checkpoints
strata serve: decode expert cache hit rate: 75.6% (51672 hits / 68332 lookups)
strata serve: KV streaming: 98.90% of 22380 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 1 windows, 2 of 2 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 366 ms (73.8 tok/s), 128 generated in 2254 ms (56.8 tok/s), drafts accepted 70 of 124, 1 checkpoints
strata serve: decode expert cache hit rate: 90.8% (79089 hits / 87070 lookups)
strata serve: KV streaming: 99.10% of 25866 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 428 ms (51.4 tok/s), 128 generated in 2070 ms (61.8 tok/s), drafts accepted 73 of 120, 1 checkpoints
strata serve: decode expert cache hit rate: 90.4% (75569 hits / 83594 lookups)
strata serve: KV streaming: 99.04% of 23808 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 415 ms (82.0 tok/s), 128 generated in 1755 ms (72.9 tok/s), drafts accepted 87 of 99, 1 checkpoints
strata serve: decode expert cache hit rate: 75.2% (50603 hits / 67286 lookups)
strata serve: KV streaming: 98.88% of 21918 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 2 windows, 4 of 5 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 370 ms (73.1 tok/s), 128 generated in 2073 ms (61.8 tok/s), drafts accepted 62 of 96, 1 checkpoints
strata serve: decode expert cache hit rate: 91.2% (70888 hits / 77711 lookups)
strata serve: KV streaming: 98.93% of 22440 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 305 ms (72.2 tok/s), 128 generated in 1971 ms (64.9 tok/s), drafts accepted 69 of 97, 1 checkpoints
strata serve: decode expert cache hit rate: 90.0% (67306 hits / 74786 lookups)
strata serve: KV streaming: 98.91% of 20928 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 420 ms (81.0 tok/s), 128 generated in 1833 ms (69.8 tok/s), drafts accepted 89 of 108, 1 checkpoints
strata serve: decode expert cache hit rate: 76.3% (52834 hits / 69265 lookups)
strata serve: KV streaming: 98.91% of 22596 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 1 windows, 3 of 3 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 358 ms (75.4 tok/s), 128 generated in 2381 ms (53.8 tok/s), drafts accepted 63 of 138, 1 checkpoints
strata serve: decode expert cache hit rate: 92.3% (89689 hits / 97169 lookups)
strata serve: KV streaming: 99.18% of 28644 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 314 ms (70.1 tok/s), 128 generated in 2112 ms (60.6 tok/s), drafts accepted 72 of 118, 1 checkpoints
strata serve: decode expert cache hit rate: 90.1% (75246 hits / 83542 lookups)
strata serve: KV streaming: 99.04% of 23706 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 410 ms (82.9 tok/s), 128 generated in 1720 ms (74.4 tok/s), drafts accepted 86 of 93, 1 checkpoints
strata serve: decode expert cache hit rate: 74.2% (48120 hits / 64866 lookups)
strata serve: KV streaming: 98.84% of 21222 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 1 windows, 2 of 3 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 355 ms (76.1 tok/s), 128 generated in 2063 ms (62.1 tok/s), drafts accepted 66 of 102, 1 checkpoints
strata serve: decode expert cache hit rate: 89.7% (70552 hits / 78635 lookups)
strata serve: KV streaming: 98.99% of 23058 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 313 ms (70.3 tok/s), 128 generated in 2014 ms (63.5 tok/s), drafts accepted 72 of 102, 1 checkpoints
strata serve: decode expert cache hit rate: 89.4% (68590 hits / 76698 lookups)
strata serve: KV streaming: 98.97% of 22104 block reads hit VRAM, 0.9 MiB read from RAM
strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: /home/mike/Programs/Strata-data/packs/swift-iq3_xxs experts (largest blob 2.33 MB), token embedding type 21 in mapped host memory (260 MiB)
strata generate: 1466 MiB of weights loaded from /home/mike/Programs/Strata-data/packs/swift-iq3_xxs (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 1781.11 MiB of weights
strata generate: KV streaming: 32768 of 131072 cells per QSA layer in VRAM, the K/V in 0.84 GiB of pinned RAM
strata generate: PLE on, table 320001536 rows of /home/mike/Programs/Strata-data/models/swift-IQ3_XXS/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf
strata generate: layer split: CUDA1 holds its weights, session and head; 11.01 GiB free
strata mtp: draft layer loaded, 820 MiB of VRAM (experts 675, dense 111)
strata generate: expert arena: mlock failed (raise ulimit -l); cudaHostRegister limited to 8 GiB for CUDA1; 10 slices pinned (7 GiB); MAP_HUGETLB unavailable (no hugetlb pool configured?); using 4 KB pages
strata generate: loaded 39.97 GiB at 4.43 GiB/s
strata generate: profile /home/mike/Programs/Strata/data/expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: layer split auto: CUDA0 36 SMs at 2.60 GHz -> 0.77 ms per layer, 8.72 GiB for experts
strata generate: layer split auto: CUDA1 36 SMs at 2.60 GHz -> 0.77 ms per layer, 7.03 GiB for experts
strata generate: layer split auto: K=26 - predicted 42.4 ms per decode window; the caches hold 9809 of 24576 profiled pairs (~97.3% of the routed mass)
strata generate: expert cache auto: 10.89 GiB free, 700 MiB reserved -> 4017 slots
strata generate: expert cache 5990 slots, 8.71 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
so a reply can differ slightly from a run without the cache (same quality:
bench/results/2026-09-27-cache-parity).
PROFILE, ranked by routing frequency, no eviction.
strata generate: pre-filled 5990 of 5990 slots from the profile; slot 0 verified
strata generate: layer split: CUDA1 runs layers 26-47, expert cache 3817 slots (7.02 GiB), 3817 of its 11264 profiled pairs; slot 0 verified
strata generate: layer split: CUDA0 runs layers 0-25
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 13 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.22)
strata generate: token graph hit path: 9807 resident experts, decided on the device
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
strata verify: window up to 6 tokens, 69.6 MiB of device buffers
strata serve: layer split: layers 0-25 (CUDA0), 26-47 (CUDA1), one hand-off per window
strata verify: window up to 6 tokens, 69.6 MiB of device buffers
strata mtp: draft head over 40525 tokens (68.0 MiB)
strata serve: 353 MiB of VRAM free with everything loaded
strata verify: captured the 6-token window (upload no error, sync no error)
strata verify: captured the 6-token window (upload no error, sync no error)
strata verify: captured the 3-token window (upload no error, sync no error)
strata verify: captured the 3-token window (upload no error, sync no error)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: captured the 2-token window (upload no error, sync no error)
strata verify: captured the 2-token window (upload no error, sync no error)
strata serve: prompt 34 tokens = 0 reused + 34 read in 498 ms (68.2 tok/s), 128 generated in 1859 ms (68.9 tok/s), drafts accepted 86 of 94, 1 checkpoints
strata serve: decode expert cache hit rate: 66.2% (43190 hits / 65193 lookups)
strata serve: KV streaming: 98.85% of 21378 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 2 windows, 5 of 7 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 416 ms (64.9 tok/s), 128 generated in 2239 ms (57.2 tok/s), drafts accepted 58 of 95, 1 checkpoints
strata serve: decode expert cache hit rate: 83.6% (66065 hits / 78996 lookups)
strata serve: KV streaming: 99.00% of 23394 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 330 ms (66.7 tok/s), 128 generated in 2137 ms (59.9 tok/s), drafts accepted 71 of 101, 1 checkpoints
strata serve: decode expert cache hit rate: 77.0% (58874 hits / 76453 lookups)
strata serve: KV streaming: 98.96% of 21996 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 460 ms (73.9 tok/s), 128 generated in 1746 ms (73.3 tok/s), drafts accepted 86 of 94, 1 checkpoints
strata serve: decode expert cache hit rate: 76.6% (50078 hits / 65384 lookups)
strata serve: KV streaming: 98.85% of 21324 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 1 windows, 1 of 2 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 361 ms (74.9 tok/s), 128 generated in 2138 ms (59.9 tok/s), drafts accepted 61 of 95, 1 checkpoints
strata serve: decode expert cache hit rate: 91.5% (71571 hits / 78181 lookups)
strata serve: KV streaming: 98.95% of 22932 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 324 ms (67.8 tok/s), 128 generated in 2014 ms (63.6 tok/s), drafts accepted 71 of 101, 1 checkpoints
strata serve: decode expert cache hit rate: 88.0% (67475 hits / 76675 lookups)
strata serve: KV streaming: 98.96% of 21882 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 426 ms (79.9 tok/s), 128 generated in 1749 ms (73.2 tok/s), drafts accepted 87 of 95, 1 checkpoints
strata serve: decode expert cache hit rate: 76.1% (49746 hits / 65370 lookups)
strata serve: KV streaming: 98.85% of 21354 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: prompt 27 tokens = 0 reused + 27 read in 371 ms (72.8 tok/s), 128 generated in 2139 ms (59.8 tok/s), drafts accepted 57 of 89, 1 checkpoints
strata serve: decode expert cache hit rate: 91.2% (70461 hits / 77222 lookups)
strata serve: KV streaming: 98.98% of 22896 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 308 ms (71.5 tok/s), 128 generated in 1947 ms (65.7 tok/s), drafts accepted 72 of 90, 1 checkpoints
strata serve: decode expert cache hit rate: 89.2% (62871 hits / 70460 lookups)
strata serve: KV streaming: 98.84% of 19578 block reads hit VRAM, 0.9 MiB read from RAM
strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: /home/mike/Programs/Strata-data/packs/swift-iq3_xxs experts (largest blob 2.33 MB), token embedding type 21 in mapped host memory (260 MiB)
strata generate: 1466 MiB of weights loaded from /home/mike/Programs/Strata-data/packs/swift-iq3_xxs (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 1781.11 MiB of weights
strata generate: KV streaming: 32768 of 131072 cells per QSA layer in VRAM, the K/V in 0.84 GiB of pinned RAM
strata generate: PLE on, table 320001536 rows of /home/mike/Programs/Strata-data/models/swift-IQ3_XXS/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf
strata generate: layer split: CUDA1 holds its weights, session and head; 11.01 GiB free
strata mtp: draft layer loaded, 820 MiB of VRAM (experts 675, dense 111)
strata generate: expert arena: mlock failed (raise ulimit -l); cudaHostRegister limited to 8 GiB for CUDA1; 10 slices pinned (7 GiB); MAP_HUGETLB unavailable (no hugetlb pool configured?); using 4 KB pages
strata generate: loaded 39.97 GiB at 4.52 GiB/s
strata generate: profile /home/mike/Programs/Strata/data/expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: layer split auto: CUDA0 36 SMs at 2.60 GHz -> 0.77 ms per layer, 8.72 GiB for experts
strata generate: layer split auto: CUDA1 36 SMs at 2.60 GHz -> 0.77 ms per layer, 7.03 GiB for experts
strata generate: layer split auto: K=26 - predicted 42.4 ms per decode window; the caches hold 9809 of 24576 profiled pairs (~97.3% of the routed mass)
strata generate: expert cache auto: 10.89 GiB free, 700 MiB reserved -> 4017 slots
strata generate: expert cache 5990 slots, 8.71 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
so a reply can differ slightly from a run without the cache (same quality:
bench/results/2026-09-27-cache-parity).
PROFILE, ranked by routing frequency, no eviction.
strata generate: pre-filled 5990 of 5990 slots from the profile; slot 0 verified
strata generate: layer split: CUDA1 runs layers 26-47, expert cache 3817 slots (7.02 GiB), 3817 of its 11264 profiled pairs; slot 0 verified
strata generate: layer split: CUDA0 runs layers 0-25
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 9 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.22)
strata generate: token graph hit path: 9807 resident experts, decided on the device
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
strata verify: window up to 6 tokens, 69.6 MiB of device buffers
strata serve: layer split: layers 0-25 (CUDA0), 26-47 (CUDA1), one hand-off per window
strata verify: window up to 6 tokens, 69.6 MiB of device buffers
strata mtp: draft head over 40525 tokens (68.0 MiB)
strata serve: 352 MiB of VRAM free with everything loaded
strata verify: captured the 6-token window (upload no error, sync no error)
strata verify: captured the 6-token window (upload no error, sync no error)
strata verify: captured the 3-token window (upload no error, sync no error)
strata verify: captured the 3-token window (upload no error, sync no error)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: captured the 2-token window (upload no error, sync no error)
strata verify: captured the 2-token window (upload no error, sync no error)
strata serve: prompt 34 tokens = 0 reused + 34 read in 490 ms (69.4 tok/s), 128 generated in 2006 ms (63.8 tok/s), drafts accepted 85 of 96, 1 checkpoints
strata serve: decode expert cache hit rate: 66.1% (44018 hits / 66639 lookups)
strata serve: KV streaming: 98.87% of 21846 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 2 windows, 4 of 4 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 418 ms (64.6 tok/s), 128 generated in 2153 ms (59.5 tok/s), drafts accepted 68 of 94, 1 checkpoints
strata serve: decode expert cache hit rate: 79.5% (59671 hits / 75099 lookups)
strata serve: KV streaming: 98.92% of 22296 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 349 ms (63.0 tok/s), 128 generated in 2175 ms (58.8 tok/s), drafts accepted 70 of 101, 1 checkpoints
strata serve: decode expert cache hit rate: 77.6% (59654 hits / 76901 lookups)
strata serve: KV streaming: 98.97% of 22230 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 418 ms (81.4 tok/s), 128 generated in 1774 ms (72.1 tok/s), drafts accepted 86 of 95, 1 checkpoints
strata serve: decode expert cache hit rate: 77.5% (51045 hits / 65875 lookups)
strata serve: KV streaming: 98.85% of 21480 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 2 windows, 4 of 6 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 361 ms (74.8 tok/s), 128 generated in 1998 ms (64.1 tok/s), drafts accepted 65 of 90, 1 checkpoints
strata serve: decode expert cache hit rate: 90.4% (66359 hits / 73385 lookups)
strata serve: KV streaming: 98.92% of 21612 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 316 ms (69.7 tok/s), 128 generated in 2039 ms (62.8 tok/s), drafts accepted 70 of 103, 1 checkpoints
strata serve: decode expert cache hit rate: 87.5% (68344 hits / 78098 lookups)
strata serve: KV streaming: 98.98% of 22374 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 419 ms (81.0 tok/s), 128 generated in 1790 ms (71.5 tok/s), drafts accepted 86 of 95, 1 checkpoints
strata serve: decode expert cache hit rate: 76.2% (50162 hits / 65853 lookups)
strata serve: KV streaming: 98.85% of 21408 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 1 windows, 2 of 2 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 359 ms (75.2 tok/s), 128 generated in 2186 ms (58.5 tok/s), drafts accepted 59 of 95, 1 checkpoints
strata serve: decode expert cache hit rate: 91.0% (72036 hits / 79133 lookups)
strata serve: KV streaming: 99.00% of 23490 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 311 ms (70.7 tok/s), 128 generated in 1917 ms (66.8 tok/s), drafts accepted 72 of 94, 1 checkpoints
strata serve: decode expert cache hit rate: 88.8% (64251 hits / 72380 lookups)
strata serve: KV streaming: 98.88% of 20340 block reads hit VRAM, 0.9 MiB read from RAM
strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: /home/mike/Programs/Strata-data/packs/swift-iq3_xxs experts (largest blob 2.33 MB), token embedding type 21 in mapped host memory (260 MiB)
strata generate: 1466 MiB of weights loaded from /home/mike/Programs/Strata-data/packs/swift-iq3_xxs (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 1781.11 MiB of weights
strata generate: KV streaming: 32768 of 131072 cells per QSA layer in VRAM, the K/V in 0.84 GiB of pinned RAM
strata generate: PLE on, table 320001536 rows of /home/mike/Programs/Strata-data/models/swift-IQ3_XXS/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf
strata generate: layer split: CUDA1 holds its weights, session and head; 11.01 GiB free
strata mtp: draft layer loaded, 820 MiB of VRAM (experts 675, dense 111)
strata generate: expert arena: mlock failed (raise ulimit -l); cudaHostRegister limited to 8 GiB for CUDA1; 10 slices pinned (7 GiB); MAP_HUGETLB unavailable (no hugetlb pool configured?); using 4 KB pages
strata generate: loaded 39.97 GiB at 4.30 GiB/s
strata generate: profile /home/mike/Programs/Strata/data/expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: layer split auto: CUDA0 36 SMs at 2.60 GHz -> 0.77 ms per layer, 8.73 GiB for experts
strata generate: layer split auto: CUDA1 36 SMs at 2.60 GHz -> 0.77 ms per layer, 7.03 GiB for experts
strata generate: layer split auto: K=26 - predicted 42.3 ms per decode window; the caches hold 9813 of 24576 profiled pairs (~97.3% of the routed mass)
strata generate: expert cache auto: 10.89 GiB free, 700 MiB reserved -> 4020 slots
strata generate: expert cache 5994 slots, 8.72 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
so a reply can differ slightly from a run without the cache (same quality:
bench/results/2026-09-27-cache-parity).
PROFILE, ranked by routing frequency, no eviction.
strata generate: pre-filled 5994 of 5994 slots from the profile; slot 0 verified
strata generate: layer split: CUDA1 runs layers 26-47, expert cache 3817 slots (7.02 GiB), 3817 of its 11264 profiled pairs; slot 0 verified
strata generate: layer split: CUDA0 runs layers 0-25
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 6 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.22)
strata generate: token graph hit path: 9811 resident experts, decided on the device
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
strata verify: window up to 6 tokens, 69.6 MiB of device buffers
strata serve: layer split: layers 0-25 (CUDA0), 26-47 (CUDA1), one hand-off per window
strata verify: window up to 6 tokens, 69.6 MiB of device buffers
strata mtp: draft head over 40525 tokens (68.0 MiB)
strata serve: 352 MiB of VRAM free with everything loaded
strata verify: captured the 6-token window (upload no error, sync no error)
strata verify: captured the 6-token window (upload no error, sync no error)
strata verify: captured the 3-token window (upload no error, sync no error)
strata verify: captured the 3-token window (upload no error, sync no error)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: captured the 2-token window (upload no error, sync no error)
strata verify: captured the 2-token window (upload no error, sync no error)
strata serve: prompt 34 tokens = 0 reused + 34 read in 533 ms (63.8 tok/s), 128 generated in 2167 ms (59.1 tok/s), drafts accepted 87 of 90, 1 checkpoints
strata serve: decode expert cache hit rate: 65.5% (41147 hits / 62825 lookups)
strata serve: KV streaming: 98.80% of 20544 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 2 windows, 4 of 4 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 463 ms (58.3 tok/s), 128 generated in 2454 ms (52.1 tok/s), drafts accepted 54 of 84, 1 checkpoints
strata serve: decode expert cache hit rate: 83.7% (63345 hits / 75669 lookups)
strata serve: KV streaming: 98.95% of 22194 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 384 ms (57.2 tok/s), 128 generated in 2404 ms (53.3 tok/s), drafts accepted 70 of 104, 1 checkpoints
strata serve: decode expert cache hit rate: 76.9% (60279 hits / 78371 lookups)
strata serve: KV streaming: 98.99% of 22584 block reads hit VRAM, 0.9 MiB read from RAM
strata verify: captured the 5-token window (upload no error, sync no error)
strata verify: captured the 5-token window (upload no error, sync no error)
strata serve: prompt 34 tokens = 0 reused + 34 read in 448 ms (76.0 tok/s), 128 generated in 2056 ms (62.3 tok/s), drafts accepted 88 of 99, 1 checkpoints
strata serve: decode expert cache hit rate: 75.3% (51012 hits / 67740 lookups)
strata serve: KV streaming: 98.90% of 22266 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 2 windows, 4 of 6 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 383 ms (70.5 tok/s), 128 generated in 2077 ms (61.6 tok/s), drafts accepted 63 of 87, 1 checkpoints
strata serve: decode expert cache hit rate: 91.5% (67124 hits / 73390 lookups)
strata serve: KV streaming: 98.92% of 21738 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 336 ms (65.5 tok/s), 128 generated in 2078 ms (61.6 tok/s), drafts accepted 70 of 98, 1 checkpoints
strata serve: decode expert cache hit rate: 88.2% (66796 hits / 75735 lookups)
strata serve: KV streaming: 98.94% of 21576 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 34 tokens = 0 reused + 34 read in 445 ms (76.3 tok/s), 128 generated in 2017 ms (63.5 tok/s), drafts accepted 86 of 98, 1 checkpoints
strata serve: decode expert cache hit rate: 75.6% (50815 hits / 67256 lookups)
strata serve: KV streaming: 98.87% of 21816 block reads hit VRAM, 1.0 MiB read from RAM
strata serve: suffix drafts: 1 windows, 3 of 5 drafts accepted
strata serve: prompt 27 tokens = 0 reused + 27 read in 385 ms (70.2 tok/s), 128 generated in 2147 ms (59.6 tok/s), drafts accepted 63 of 94, 1 checkpoints
strata serve: decode expert cache hit rate: 89.8% (68479 hits / 76251 lookups)
strata serve: KV streaming: 98.97% of 22758 block reads hit VRAM, 0.9 MiB read from RAM
strata serve: prompt 22 tokens = 0 reused + 22 read in 331 ms (66.4 tok/s), 128 generated in 2004 ms (63.9 tok/s), drafts accepted 74 of 96, 1 checkpoints
strata serve: decode expert cache hit rate: 88.4% (64845 hits / 73331 lookups)
strata serve: KV streaming: 98.91% of 20832 block reads hit VRAM, 0.9 MiB read from RAM
strata generate: layer split across 3 GPUs: CUDA0, then CUDA1,2 (split auto)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: /home/mike/Programs/Strata-data/packs/swift-iq3_xxs experts (largest blob 2.33 MB), token embedding type 21 in mapped host memory (260 MiB)
strata generate: 1466 MiB of weights loaded from /home/mike/Programs/Strata-data/packs/swift-iq3_xxs (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 1781.11 MiB of weights
strata generate: KV streaming: 32768 of 131072 cells per QSA layer in VRAM, the K/V in 0.84 GiB of pinned RAM
strata generate: PLE on, table 320001536 rows of /home/mike/Programs/Strata-data/models/swift-IQ3_XXS/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf
strata generate: layer split: CUDA1 holds its weights, session; 11.42 GiB free
kv_stream: reset: no kernel image is available for execution on the device
Console log
Strata - Qwen3.8-Flash-Next on a normal PC (NVIDIA GPU + system RAM + CPU)
Tuning Strata for this PC: the output speed is measured with a few engine settings (the PCIe share, the
draft depth, the CPU threads). It takes about 5-10 minutes; the PC is busy meanwhile.
Loading the model for the measurements ...
[strata] starting the engine: reading the model's weights ...
[strata] loading the experts into RAM (about 40 GB) and locking part of them for the GPU.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the browser opens when it is ready.
[strata] experts loaded: 39.97 GiB at 4.53 GiB/s (14 s so far)
[strata] filling the GPU's expert cache (5990 experts, 8.71 GiB of VRAM) ...
[strata] almost ready ...
PCIe share 0.00: 62.2 tok/s
PCIe share 0.20: 63.9 tok/s
PCIe share 0.35: 60.0 tok/s
PCIe share 0.55: 60.5 tok/s
PCIe share 0.75: 60.3 tok/s
draft floor 0.30: 59.5 tok/s
draft floor 0.50: 63.2 tok/s
draft floor 0.70: 66.5 tok/s
Measuring with 13 CPU workers (restarts the engine) ...
[strata] starting the engine: reading the model's weights ...
[strata] loading the experts into RAM (about 40 GB) and locking part of them for the GPU.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the browser opens when it is ready.
[strata] experts loaded: 39.97 GiB at 4.43 GiB/s (13 s so far)
[strata] filling the GPU's expert cache (5990 experts, 8.71 GiB of VRAM) ...
[strata] almost ready ...
13 workers: 64.6 tok/s
Measuring with 9 CPU workers (restarts the engine) ...
[strata] starting the engine: reading the model's weights ...
[strata] loading the experts into RAM (about 40 GB) and locking part of them for the GPU.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the browser opens when it is ready.
[strata] experts loaded: 39.97 GiB at 4.52 GiB/s (13 s so far)
[strata] filling the GPU's expert cache (5990 experts, 8.71 GiB of VRAM) ...
[strata] almost ready ...
9 workers: 65.4 tok/s
Measuring with 6 CPU workers (restarts the engine) ...
[strata] starting the engine: reading the model's weights ...
[strata] loading the experts into RAM (about 40 GB) and locking part of them for the GPU.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the browser opens when it is ready.
[strata] experts loaded: 39.97 GiB at 4.30 GiB/s (14 s so far)
[strata] filling the GPU's expert cache (5994 experts, 8.72 GiB of VRAM) ...
[strata] almost ready ...
6 workers: 62.6 tok/s
[ok] tuned for this PC: --pcie-frac 0.20, --spec-min-p 0.70 (64.6 tok/s)
[ok] GPUs: GPU 0 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 1 (NVIDIA GeForce RTX 5060 Ti, 16 GB) + GPU 2 (NVIDIA GeForce RTX 3060, 12 GB) together (layers split auto)
----------------------------------------------------------------------------------------------------
Starting swift-1.5-iq3_xxs: it loads about 40 GB into RAM and locks part of it for the GPU.
While it does, YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES (longer the first time after a
restart). That is normal: please wait and don't close this window - the browser opens when it is ready.
Later, closing this window stops the model.
----------------------------------------------------------------------------------------------------
loading the model (the first start takes a minute or two) ...
[strata] layer split across GPUs [0, 1, 2] (auto)
[strata] starting the engine: reading the model's weights ...
[strata] loading the experts into RAM (about 40 GB) and locking part of them for the GPU.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the browser opens when it is ready.
Traceback (most recent call last):
File "/home/mike/Programs/Strata/serve/server.py", line 1616, in <module>
sys.exit(main())
^^^^^^
File "/home/mike/Programs/Strata/serve/server.py", line 1548, in main
engine = StrataEngine(cfg["exe"], engine_args(cfg), cwd=cfg.get("cwd"), log=cfg.get("log"), env=env)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mike/Programs/Strata/serve/server.py", line 182, in __init__
raise RuntimeError("the engine exited before it was ready" + (f" (see {log})" if log else ""))
RuntimeError: the engine exited before it was ready (see /home/mike/Programs/Strata/strata-swift-iq3_xxs.log)
Assisted with Qwen3.8 27b
Activity
Niko1221 commentedon Sep 29, 2026
Thanks @pdsmike, confirmed on 0.1.26: setup never checks that the installed engine has code for every card in
--gpus, and even--setupskips the rebuild when the source hasn't changed. We'll make both paths recompile for all the chosen cards. Meanwhile, deletingengine/BUILD.jsonand running--setup --gpus 0,1,2forces a compile for all three.setup: AVX2 from the CPU itself when Windows says no (#159); the engi…
Niko1221 commentedon Sep 29, 2026
Fixed in 0.1.27: setup now checks that the engine has code for every card the model runs on, both with
--setup --gpus ...and when starting. If a card is missing, it compiles the engine for all the chosen cards and keeps the generations the engine was built for. A ready-made engine that lacks a card is replaced by a compiled one. Thanks @pdsmike! Please reopen if a mixed set still fails to start.setup: AVX2 from the CPU itself when Windows says no (Niko1221#159)
setup: AVX2 from the CPU itself when Windows says no (Niko1221#159); …
setup: AVX2 from the CPU itself when Windows says no (Niko1221#159); …
cuda: the non-TS reduction tails as else branches of if constexpr (TS…