Repository navigation
Arena mode, one GPU on Windows: ~3 ms per window in the blocking adaptive tier, and layers 32-47 never get a PCIe share (RTX 3090 + Ryzen 5 5600X) #1903
Description
Machine. RTX 3090 24 GB (sm_86, 350 W, also drives the desktop), PCIe 4.0 x16, Ryzen 5 5600X (6 cores, AVX2), 2 x 32 GB
DDR4-3600, Windows 11. Qwen3.8-Flash-Next GSQ-RCO IQ3_S, --max-context 262144 --kv int8 --kv-grow --vram-reserve-mib 1200,
--spec 4 --mtp, calibrated adaptive tier (--adapt-every 1 --adapt-swaps 80 --adapt-decay 0.97). The default arena: every
expert in RAM, 8,289 expert slots in VRAM. Strata v0.1.42 (61b3fb5d); local builds with VS 2022 17.14 + CUDA 12.8, sm_86.
1. The blocking adaptive tier: ~3.2 ms of a 25.9 ms window (same hole as #1863)
STRATA_DECODE_TIMING=1, a cold short prompt: 25.9 ms per window = verify 21.0 + commit/emit 0.1 + draft 1.6, so ~3.2 ms
are outside the breakdown, as #1863 measured on its machine. In the arena mode the tier's thread is created and joined
every window (generate.cpp ~11630-11685) and the next window waits for its copies (apply_pending(true) →
cudaEventSynchronize(adapt_ev), then a synchronous res_upload). About a third of the swaps come from the part of the
arena that is not page-locked (below), so their cudaMemcpyAsync is synchronous. STRATA_ADAPT_LAG=2 does not help here
(it halves the swap rate: −8 % cold). --adapt-async 1 would, but it is refused outside the resident RAM mode.
2. Windows page-locks the first ~30 GiB of the arena only: layers 32-47 get no PCIe share
cudaHostRegister of the whole 47 GiB arena fails (WDDM shared budget, about half the RAM), the sliced fallback registers
the first 32 slices (30 GiB, layers 0-31), and pcie_layer() is false for layers 32-47: their misses are 100 % CPU, while
~10 GiB of the registered prefix hold experts that are GPU-resident and never read by DMA. Also m = (nmiss * pcie_num) >> 8
rounds the PCIe share to 0 at 1-2 misses per layer, which is the common case at a 88-95 % hit rate.
Measured (cold A/B, paired)
12 distinct short prompts (FR/EN, code, prose), each sent once per fresh engine with a fixed seed, temperature 0.7, 256
tokens, 2 rounds with the arms' order reversed; per-prompt ratio against the reference arm.
| Arm (local builds of the PR branches against the v0.1.42 tag) | Decode tok/s (median) | vs v0.1.42 | Prompts faster |
|---|---|---|---|
| v0.1.42 | 97.2 | 1.00 | — |
--adapt-async 1 allowed in the arena (copy-in only) |
107.7 | x1.07 (x1.11 in an earlier run) | 12/12 |
pin by need + STRATA_PCIE_MIN1=1 + --pcie-frac 0.6, blocking tier |
99.3 | x1.00 | 6/12 |
the same on top of --adapt-async 1 and other opt-in switches |
— | x1.02-1.03 | 10/12 |
With --adapt-async 1 the window drops to 21.6 ms (adapt wait 0.41 ms, join 0). Pin by need: 28.96 GiB registered = 92 % of
the non-resident experts, every layer gets a PCIe share; the prompt read is 5-7 % faster at 56K-117K tokens (2,172 → 2,313
and 2,091 → 2,208 tok/s).
A pitfall for anyone measuring this: a prompt sent twice in a row runs ~25 % faster the second time (the tier has adapted to
it); an A/B that always sends arm B second shows a gain that is not there.
What I will send
Two pull requests, each opt-in with the default path byte-identical to v0.1.42:
- the asynchronous adaptive tier in the arena mode (one GPU), with the K/V-grow and
VRAMfences it needs; - pin by need (
STRATA_PIN_BY_NEED=1, page-aligned arena stride) and the PCIe share floor (STRATA_PCIE_MIN1=1).
Raw data, harness and the full write-up: https://github.com/RubenTorney/Strata/tree/perf/async-adapt-arena
(bench/results/2026-10-11-rtx3090-5600x-async-arena); each PR carries its own report.
Not tested: HIP and SYCL builds, Linux, layer splits / several GPUs, the resident RAM mode with these builds (the new
paths are gated on the arena and one GPU; the K/V-grow fence also applies to the resident mode's asynchronous tier).
Activity
batch A/B/C: SIMT scorer adopted, the rest measured and left off; sto…
qiuzan2001 commentedon Oct 15, 2026
A measurement of
--adapt-async 1on another machine: no change beyond the noise here.Machine: RTX 5070 Ti 16 GB (PCIe 4.0 x16), Ryzen 7 5800X3D (AVX2, no AVX-512), 80 GiB DDR4-3200, Ubuntu 26.04, driver 595.99.02, CUDA 13.1. Strata 0.1.44.1 source build (
296b59d2), Swift 1.5 IQ3_S in the expert arena, setup's settings with a 1,048,576-token YaRN context (int8 KV, 32,768 cells resident,--spec 4 --spec-min-p 0.5, MTP), 3,124-3,126 experts in VRAM. The only difference between the arms is--adapt-async 1(the engine logsadaptive tier asynchronous (--adapt-async 1): up to 96 swaps per round, a round every 4 windows (arena: copy-in only)).Method: a fresh server per arm and round, 4 rounds in the order AB, BA, AB, BA, one discarded warm-up answer per start. Each round: eight different short prompts (code, prose and lists) and one 6,345-token prompt, 256 new tokens each, greedy, thinking off. "Pooled" = sum of the engine's
predicted_n/ sum ofpredicted_msover the eight short answers of a round.--adapt-async 1Drafts accepted over all short answers: 4,825 of 6,986 (default) and 4,764 of 7,063 (async); 4 of the 36 answer pairs are byte-identical, so the arms mostly wrote different text. On this card and model the paired changes (-1.0% to +2.3%) are inside the run-to-run spread, unlike the +4-5% the maintainer measured on an RTX 5070 with Q2_0 (#1904). Not tested: another model size, a shorter context (more experts in VRAM), long generations,
--batch, orSTRATA_DECODE_TIMING=1's adapt wait times. Per-request data on request.qiuzan2001 commentedon Oct 16, 2026
A measurement of what the blocking tier costs per window on another PC, in case it helps as a second data point. RTX 5070 Ti 16 GB (PCIe 4.0 x16), Ryzen 7 5800X3D, 80 GiB DDR4-3200, Ubuntu, Strata 0.1.44.1, Swift 1.5 IQ3_S in the arena,
--adapt-every 1 --adapt-swaps 80 --adapt-decay 0.97: a short-chat decode window of 32.3 ms = GPU 10.4 + CPU experts 17.3 + draft 2.2 + adapt wait 1.6 + other 0.8 ms (from Strata'sSTRATA_DECODE_TIMINGlines, last request). So here the tier's wait is about 5 % of the window, against your ~12 % (3.2 of 25.9 ms) on Windows, which fits your point that the non-page-locked part of the arena makes the copies synchronous there.An A/B that bears on the swap count: two profile runs on this box that differed only in
--adapt-swaps 160against80(same model, settings and prompts): agent turns (3K to 15K tokens of context, six turns) +5.1 % with 80, 6 of 6 turns faster, sign test p = 0.031; short chats and the 16K prompt inside the noise. I measured it with Strata Profile Studio, which draws the window split per request and the before/after per turn.Niko1221 commentedon Oct 19, 2026
Thanks for taking the time to report this.
We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to read properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.
To reopen it takes about two minutes:
/reopen. A check runs and either reopens the issue or lists what is still missing.The rules are in CONTRIBUTING.md.