[RubenTorney]

Title

Issue: #1903

Summary

In the arena mode the blocking adaptive tier creates and joins a thread every verify window and the next window waits for its
copies: ~3.2 of 25.9 ms per window on this machine (the interval #1863 found outside the timing breakdown). This lets
--adapt-async 1 run in the arena too. The arena holds every expert in RAM, so a round is a copy-in only: no copy back, no RAM
move, no exchange buffers. Opt-in; without --adapt-async 1 (and in the resident RAM mode, and on several GPUs) nothing changes.

Cold A/B against the v0.1.42 tag (12 distinct prompts x 2 rounds, fresh server per arm and round, paired): x1.07, faster on
12 of 12 prompts (97.2 → 107.7 tok/s; x1.11 in an earlier run the same day, when the learned profile was younger and the tier
swapped more). One cold window: 25.9 → 21.6 ms, adapt wait 0.41 ms, join 0.

What changed

• src/program/generate.cpp
  • The --adapt-async gates accept the arena (srcp == &arena_src) on one GPU; a layer split stays on the blocking tier.
  • The asynchronous tier's state machine, arena case: exchange = false, the source blob is ArenaExpertSource::blob_at(),
    the RAM-move steps are skipped. The evicted slot is marked non-resident and res_upload runs before the round's copies are
    posted; an expert becomes resident only once its copy's event has completed.
  • STRATA_ADAPT_ASYNC_WAIT=1 (default in the arena): the tick after a round waits for that round, so the window in which an
    expert turns GPU-resident does not depend on timing. STRATA_ADAPT_PICK_INLINE=1 (default in the arena): the pick runs on
    the host thread, one round per window. STRATA_ADAPT_ASYNC_BATCH=N caps the copies posted per tick (default 0 = all;
    16 measured 16 % slower here). Arena rounds are as large as the blocking tier's (--adapt-swaps, not capped at 96).
  • Fences: every K/V grow (kvg_ensure's quiesce lambdas), kv_quiesce and the VRAM command now land the round in flight
    (adapt_quiesce) — they only called apply_pending(true), so a swap still copying could write a slot just lent to the K/V.
    This also applies to the resident mode's asynchronous tier. A failed round still drains its copies before slots move.
  • STRATA_DECODE_TIMING=1: the serial --serve line adds adapt wait X join Y, Z swaps/window (part of fix: Complete the synchronous decode timing breakdown #1863).
  • --adapt-async help text.
• src/core/expert_source.cpp, include/strata/core/expert_source.hpp: ArenaExpertSource::blob_at(), blob() without the
  read counter (it runs on the job thread).
• docs/DETAILS.md: the arena paragraph next to --adapt-async 1.
• bench/results/2026-10-11-community-rtx3090-5600x-arena-async/: method, numbers, raw data (sha256 of every answer), harness.

How the default path was checked

• Bit-exact mode (STRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 0), greedy, 6 prompts, two fresh v0.1.42 engines and one of this branch:
  on the 5 prompts that v0.1.42 reproduces between two of its own fresh engines, v0.1.42 and this branch without
  --adapt-async give the same answers byte for byte (sha256), with --expert-cache auto and with a fixed size. The 6th
  prompt differs between two runs of v0.1.42 itself in that mode on this machine.
• Built: CUDA only (Windows, VS 2022 17.14, CUDA 12.8, sm_86). Not built: HIP, SYCL (no such hardware here); the change is
  host code in generate.cpp / expert_source.cpp, no backend-specific path.

Extra Notes

• Like the resident mode's asynchronous tier, answers are not bit-exact from run to run with the option on (which window first
  computes a swapped-in expert on the GPU). On this machine greedy answers already differ between two fresh engines of the
  unpatched build (the CPU/GPU split of the misses depends on timing).
• After a long prompt the gain shrinks: the prompt itself has moved its experts into VRAM (hit rate 97-99 %), the tier has
  little left to swap: level with v0.1.42 at 56K and 117K tokens (95.3 / 97.3 and 97.0 / 96.1 tok/s).
• Robustness (full test branch: this change plus other opt-in switches): 63K-token prompt then 4 follow-up turns, K/V grown
  to 65,536 cells during the conversation, no ERR.
• Reviewed for fence correctness before the A/B (VRAM command after STOP, K/V grow, request start drain, error path).

🤖 Generated with Claude Code

https://claude.ai/code/session_015oFHMEgBzk7kQSqUVWoc67

[Niko1221]

Thanks for this, and for the report: the method (fresh server per arm, distinct prompts, paired, order reversed) is the one we use ourselves, and the fence fix in kvg_ensure / kv_quiesce / the VRAM command is a real fix that also covers the resident-RAM async tier.

We ported it to the 0.1.44 branch with your authorship kept, as an opt-in (--adapt-async 1), with the default path unchanged. What we measured (RTX 5070 12 GB, Windows, Q2_0 arena, 5 interleaved pairs, three batches): decode +4.2 to +5.2% (13 of 15 pairs faster), with the gain on code and document prompts (+7 to +10%) and the story prompt level; on an R9700 (HIP, 20K cache slots) the tier is level except on a cold prompt (+12.9%), which matches your note that the gain follows how much the tier swaps. Default-path gates are byte-identical (10 prompts + 4K/20K residual dumps on CUDA sm_86 in the arena and in the resident mode, and on HIP), and it builds on CUDA, HIP, SYCL and Metal.

One thing we did on top, in the combination with your other PR: blob_at has to follow the padded layout of STRATA_PIN_BY_NEED, otherwise the async tier would copy the wrong blob. Answers with the tier on differ from the blocking tier (deterministic run to run, but not equal to it), so it stays opt-in until we have a decode KL with the tier running; we did not manage a meaningful one yet. It will be in the 0.1.44 notes as opt-in, with your name.