[Xiaomi2008]

Follow-up to #864 (the opt-in exchange buffer rotation). I searched the open threads: #1340 and #1838 touch the resident RAM exchanges but not this limit.

What it changes

STRATA_EXCHANGE_ROTATE=1 now works on packs whose layers differ in expert block size. Until now rotation required one block size for all layers and fell back to the copy path otherwise ("requires equal-size expert blocks"). Swift 1.5 IQ3_S has seven sizes (1,510,400 to 2,662,400 bytes) and IQ3_XXS nine, so neither could rotate.

• ExchangeStorage keeps its spare buffers per block size (initialize_sized). A swap exchanges two experts of one layer, so an exchange takes a spare of its layer's size and hands back one of the same size; every size keeps its number of spares. commit() refuses a spare of another size and an expert of another layer.
• exchange_buffer() takes the layer; reserve_exchanges() takes the number of trailing buffers that only stage copies (--adapt-async's second half), kept in the largest size alone.
• With --adapt-async 1 a rotated round flips in the tick that sees the copies in (nothing for the job to move), one verify window sooner.

What it leaves alone

Rotation stays opt-in. Unset, nothing new runs: the copy path, the uniform initialize() geometry and its checks are as before, and no default or preset changes. Switched on, the exchange allocation is one spare of every size per swap of a round: 1.06 GiB of page-locked RAM on IQ3_S with --adapt-swaps 80 (1.26 GiB with --adapt-async 1) against 0.20 and 0.40 GiB.

Measured

Core i7-13700K, RTX 5090 32 GB, 64 GB DDR5, Windows 11, driver 617.42. Swift 1.5 IQ3_S, 128K, --resident-experts (about 11,000 of 24,576 experts in VRAM), --adapt-every 1 --adapt-swaps 80 --adapt-decay 0.97, --spec 4, --pool-affinity p-cores.

This branch (main at 7695859 + this commit, engine 0.1.43), same day, one binary, only the variable differs. One 8K prompt, the same 1,500 forced tokens (--spec-follow), blocking tier, runs interleaved:

	decode tok/s
unset	171.2, 170.3, 173.2
STRATA_EXCHANGE_ROTATE=1	195.9, 194.9, 197.2

An unset run and two rotated runs that sized the cache alike (11,099 slots) wrote the same --window-hashes file for all 434 windows. (The unset run that sized 11,017 slots differs from the others, as any other cache size does.)

The same commit on 0.1.42 (my fork's main, which also carries #1911 and #1912):

Run	Rotation off	Rotation on
The same forced-token run	161.6-165.1 (five runs)	184.9-188.7 (four runs)
--serve, 23 coding-agent turns (tools/agent_turns_bench.py's turns, greedy, up to 400 new tokens), three interleaved rounds, blocking tier (off = the build before)	147.8 / 159.2 / 158.4	175.2 / 184.9 / 172.8
The same turns with --adapt-async 1, one binary, two rounds	166.0 / 170.1	185.6 / 182.5

The tier swaps about 30 experts per verify window all through a run on this box, so the copy path moved about 50 MiB of RAM per window on the decode thread.

How it was checked, and what was not

• tests/core/exchange_storage_test.cpp gains a three-size case (6,144 exchanges: bytes, aliases, the refusals, overlapping resident blocks) and passes on this branch, built alone with MSVC 14.50. I did not run it under ASan/UBSan, and I did not run file_expert_source_test --rotation-gpu.
• Built for CUDA only on this branch (sm_120, CUDA 13.0.88, MSVC 14.50 with nvcc's --allow-unsupported-compiler). HIP and SYCL were not compiled; expert_source.cpp and generate.cpp are built by both.
• Not measured: Linux, AMD cards, a layer split, a pack with one block size on this machine, the serve runs on 0.1.43.

Details and the numbers: docs/EXCHANGE_ROTATION.md.

🤖 Generated with Claude Code

[Niko1221]

Thanks, nice patch. We ported it with your authorship onto 0.1.44.1 and measured it on an RTX 3060 12 GB with the IQ3_XXS pack in resident RAM mode: decode 58.96 -> 62.03 tok/s (+5.2%, 5 of 5 interleaved pairs faster), prompt speed unchanged, and the tokens are identical to the build without it when the adaptive tier swaps (checked on a Tesla P100 with the same pack). It stays an opt-in switch for 0.1.45 because it needs about 1.4 GiB more page-locked RAM; we will look at making it the default after the combined test.

[Niko1221]

Thanks for the pull request.

Your change is already in main for 0.2.0-beta, with your credit, so there is nothing left for you to do here. We're closing it because we're resetting the tracker for 0.2.0-beta: there are more than 1,300 open issues and pull requests, and we're closing everything that isn't being worked on right now.

If part of your work did not make it in and you want it considered, open a new pull request on top of main that follows CONTRIBUTING.md, or comment /reopen after fixing the title area prefix and the template.