Start: read the expert arena unbuffered when the file cache cannot help (Windows; part 1 of #285) by sergqwer · Pull Request #357 · Niko1221/Strata

sergqwer · 2026-10-01T09:25:26Z

Part 1 of the #285 split, on 0.1.31.

A buffered read copies every expert through the file cache. That pays off only when the cache can keep the files for the next start. On a PC whose RAM cannot hold them beside the arena, every start reads the drive anyway, and the copy is pure cost. For example, a 64 GB PC with IQ2_XS has 24 GiB left for 36.5 GiB of files.

What changes:

experts_unbuffered() (pinned.cu) decides the reader.
- Timed random 64 KiB reads tell whether the files are already in the cache: a cached read takes ~10 us, the drive ~80.
- Unbuffered is chosen only when the files are not cached and the RAM left beside the arena could not keep them either.
- So a warm restart (idle unload) stays on 0.1.31's FILE* path. A cold start on a PC with enough RAM also stays buffered, so the restart after it is warm.
- STRATA_UNBUFFERED_LOAD=1/0 forces the choice. The log names the reader and why.
load_experts_gguf, unbuffered:
- each chunk's 4 KiB-aligned window is read with FILE_FLAG_NO_BUFFERING into an aligned buffer and scattered into the blobs;
- it keeps a handle per role file, as expert_gguf_file() resolves it (split shards);
- 16 readers.
load_experts_direct (pinned.cu): a pack's experts.bin ranges are read unbuffered straight into the arena. If the ranges are not 4 KiB-aligned, it falls back to load_experts_ranges.
Linux and macOS read as before: experts_unbuffered returns false there, and both unbuffered readers fall back to the buffered ones.

Greedy tokens are identical unbuffered, buffered and on 0.1.31 main.

Start times

Setup:

IQ2_XS without experts.bin, RTX 5090, PCIe 5 drive, generate of 16 tokens.
Cold: the IQ2_XS files were pushed out of the file cache by reading 142 GB of other files.
Restart: the same start again right away.
Each cell shows the time until the arena is loaded / until the 16-token answer is done.

An emulated 64 GB PC (a locked-pages RAM ballast leaves 58 GiB), large pages, two rounds:

	cold	restart
0.1.31	22.1-25.2 / 25.9-29.0 s	12.5-16.0 / 16.3-19.6 s
part 1	18.3-20.2 / 21.9-23.6 s	6.0-6.1 / 8.8 s

Here the probe finds the files uncached on every start, and 58 GiB cannot keep 36.5 GiB of files beside a 33 GiB arena. So part 1 reads unbuffered both times. 0.1.31's restart reads the drive again too: the cache lost the files to the arena.

128 GB, nothing changed, large pages:

	cold	restart
0.1.31 (3 runs)	17.6-18.7 / 21.1-22.4 s	4.6-5.1 / 7.0-7.6 s
part 1	19.3 / 23.1 s	4.7 / 7.4 s

Both of these starts read through the file cache:

On the cold start the probe found the files uncached (1 of 16 reads). But 84 GiB beside the arena can keep them, so the cache path was chosen, and the restart that followed was warm (15 of 16).
The cold spread is the drive's. Parts 1+2 gave 17.6-19.9 s cold across six runs, with the probe forced off (STRATA_UNBUFFERED_LOAD=0) as well as on.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

sergqwer · 2026-10-01T13:31:46Z

Pushed b26904d: a build fix for this PR. load_experts_gguf gained bool unbuffered = false in its definition, but the header still declared the four-parameter form. The engine's caller is in the same file, so it built; tests/core/expert_layout_test.cpp calls it through the header and failed to link (LNK2019) with STRATA_BUILD_TESTS and STRATA_NATIVE_EXPERTS on. The default now lives in the declaration; expert_layout_test passes.

#358 and #362 are stacked on this branch and are rebased onto it (their content is unchanged, apart from #358 declaring its ready parameter the same way).

🤖 Generated with Claude Code

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…lp (Windows; part 1 of #285) A buffered read copies every expert through the file cache. That pays off only when the cache can keep the files for the next start. On a PC whose RAM cannot hold them beside the arena (a 64 GB PC with IQ2_XS: 24 GiB left for 36.5 GiB of files), every start reads the drive anyway, and the copy is pure cost. - experts_unbuffered() (pinned.cu): timed random 64 KiB reads tell whether the files are in the cache already (a cached read takes ~10 us, the drive ~80). Unbuffered is chosen only when they are not and the RAM left beside the arena could not keep them either. A warm restart that 0.1.31 serves from the cache stays on its FILE* path. STRATA_UNBUFFERED_LOAD=1/0 forces the choice; the log names the reader and why. - load_experts_gguf: each chunk's 4 KiB-aligned window is read with FILE_FLAG_NO_BUFFERING into an aligned buffer and scattered into the blobs, with a handle per role file as expert_gguf_file() resolves it (split shards). 16 readers. - load_experts_direct (pinned.cu): a pack's experts.bin ranges are read unbuffered straight into the arena. When the ranges are not 4 KiB-aligned, it falls back to load_experts_ranges. IQ2_XS without experts.bin, RTX 5090, emulated 64 GB PC (a locked-pages RAM ballast leaves 58 GiB), two rounds: | | first start: arena / exit | restart: arena / exit | |---|---|---| | 0.1.31 | 22.1-25.2 s / 28.2-31.0 s | 12.5-16.0 s / 17.5-20.8 s | | this | 18.3-20.2 s / 23.1-25.4 s | 6.0-6.1 s / 9.8-9.9 s | With 128 GB the probe finds the files cached (15 of 16 reads) and the buffered path runs as before. Greedy tokens are identical unbuffered, buffered and on 0.1.31 main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

The definition gained a defaulted parameter while the header kept the four-parameter declaration. The engine's caller is in the same file, so it built, but tests/core/expert_layout_test.cpp calls it through the header and failed to link (LNK2019, unresolved load_experts_gguf). The default now lives in the declaration; expert_layout_test passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

sergqwer · 2026-10-01T18:16:30Z

Rebased on 0.1.32 (c499bd1), no conflicts.

Checked on 0.1.32: IQ2_XS (ISTA), RTX 5090, a fixed --expert-cache, --vram-reserve-mib 1500. The first generated token's logits (STRATA_DUMP_FIRST_LOGITS, now in main) are byte-identical to main's after a 2K and a 32K prompt. Every target builds with STRATA_BUILD_TESTS.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…ot keep it beside the RAM budget (Windows; #286 ported into 0.1.31's tier) With --resident-budget-gib on a PC whose RAM cannot hold the expert files beside the budget, every expert outside the budget comes from the drive anyway. Through the mapped GGUF that costs twice: - the page faults are small requests on the critical path (the prompt path's stager, the profile fill); - the pages they bring in take the RAM the budget was sized for. On an emulated 32 GB PC, the 16 GiB budget was clamped to 11.65 GiB after the profile fill. - FileExpertSource::set_unbuffered: chosen by experts_unbuffered (#357) with the budget as the RAM it keeps: whenever the files could not be kept beside it, cached now or not - a partly cached file's mapped pages would still land in the working set (seen on an emulated 64 GB PC: the budget shrank, then RAM ran out). Each shard is opened FILE_FLAG_NO_BUFFERING | OVERLAPPED. - copy_from_files, the decode prefetch and the complement copy read every role window of a batch at once, then copy it into place. - Windows that are less than 1 MiB apart merge into requests of up to 32 MiB. NTFS runs a file's unbuffered reads one at a time while the file is mapped anywhere, and the engine always maps the shards (token embedding, PLE). Measured on a PCIe 5 drive at queue depth 48: 512 KiB reads give 10.4 GB/s on an unmapped file and 3.4 GB/s on a mapped one; 8 MiB reads give 8.6 GB/s. - The profile fill reads its pairs in batches of 64, the next batch while this one is copied into the slots. - warms() is false when unbuffered. A PrefetchVirtualMemory of the mapping would read the experts through the cache, a second time. - Otherwise nothing changes: the mapped path stays for a keepable set, for no budget, and off Windows. STRATA_UNBUFFERED_LOAD=1/0 forces the choice. Greedy tokens are identical with the reads unbuffered, through the cache, and on 0.1.31. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…the GPU (#473), IQ4_XS on AVX-2 (#415), unbuffered loads on Windows (#357 #362), --peer-device (#531), a 6 GB card starts (#496), PR batch; setup requires it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

Niko1221 · 2026-10-03T01:17:55Z

Released in 0.1.38: the unbuffered arena read when the file cache can't help. Merged under your name (GitHub doesn't show it as merged because it went in by hand). Thank you!

Upstream v0.1.38: faster prompts (one gather per expert group Niko1221#372, the first chunk's PLE rows beside layer 0 Niko1221#374, DeltaNet three heads per thread Niko1221#413), --kv q4_0 prompts on tensor cores Niko1221#452, Q5_0 experts on the GPU Niko1221#473, IQ4_XS on AVX-2 Niko1221#415, unbuffered expert loading on Windows Niko1221#357/Niko1221#362, the --peer-device second-GPU expert tier Niko1221#531, a 6 GB card starts Niko1221#496, the Host header / DNS-rebinding checks, PR batch. Our side: the AVX1 floor (STRATA_ISA_FLOOR, cpu_avx1_ok, the relaxed gate, the scalar router fallback), the AVX1 router kernel (kq_avx1.cpp), the retune patch + setup hooks.

…ain once the RAM copy is built (#577) 0.1.38 made the choice before the resident set existed, from the budget asked for, against every shard's bytes (dense weights and the PLE table included). On a 96 GB PC with UD-Q4_K_XL (budget 72 GiB, 48.94 GiB really resident) that read 83.0 - 72 - 4 = 7 GiB of room against 103.7 GiB of files, so every refill after a prompt read the drive: prompts 15-40% slower than 0.1.34. - platform::file_cache_keeps(avail, arena, read): the decision, unit-tested in platform_memory_test with the reporter's numbers and the 32/64 GB PCs of #357/#362. - set_unbuffered: the arena is at most every expert's bytes, and the cache has to keep only the experts outside it. - recheck_unbuffered, after pin_cache_complement: the same decision from the RAM the copy really holds and the expert bytes outside it; switches either way (startup, nothing reading) and logs the numbers. - The 32 and 64 GB PCs still read unbuffered (the profile fill and the copy too, as before). Same tokens either way: only how the same bytes are read changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…ot keep it beside the RAM budget (Windows; #286 ported into 0.1.31's tier) With --resident-budget-gib on a PC whose RAM cannot hold the expert files beside the budget, every expert outside the budget comes from the drive anyway. Through the mapped GGUF that costs twice: - the page faults are small requests on the critical path (the prompt path's stager, the profile fill); - the pages they bring in take the RAM the budget was sized for. On an emulated 32 GB PC, the 16 GiB budget was clamped to 11.65 GiB after the profile fill. - FileExpertSource::set_unbuffered: chosen by experts_unbuffered (#357) with the budget as the RAM it keeps: whenever the files could not be kept beside it, cached now or not - a partly cached file's mapped pages would still land in the working set (seen on an emulated 64 GB PC: the budget shrank, then RAM ran out). Each shard is opened FILE_FLAG_NO_BUFFERING | OVERLAPPED. - copy_from_files, the decode prefetch and the complement copy read every role window of a batch at once, then copy it into place. - Windows that are less than 1 MiB apart merge into requests of up to 32 MiB. NTFS runs a file's unbuffered reads one at a time while the file is mapped anywhere, and the engine always maps the shards (token embedding, PLE). Measured on a PCIe 5 drive at queue depth 48: 512 KiB reads give 10.4 GB/s on an unmapped file and 3.4 GB/s on a mapped one; 8 MiB reads give 8.6 GB/s. - The profile fill reads its pairs in batches of 64, the next batch while this one is copied into the slots. - warms() is false when unbuffered. A PrefetchVirtualMemory of the mapping would read the experts through the cache, a second time. - Otherwise nothing changes: the mapped path stays for a keepable set, for no budget, and off Windows. STRATA_UNBUFFERED_LOAD=1/0 forces the choice. Greedy tokens are identical with the reads unbuffered, through the cache, and on 0.1.31.

…the GPU (#473), IQ4_XS on AVX-2 (#415), unbuffered loads on Windows (#357 #362), --peer-device (#531), a 6 GB card starts (#496), PR batch; setup requires it

…ain once the RAM copy is built (#577) 0.1.38 made the choice before the resident set existed, from the budget asked for, against every shard's bytes (dense weights and the PLE table included). On a 96 GB PC with UD-Q4_K_XL (budget 72 GiB, 48.94 GiB really resident) that read 83.0 - 72 - 4 = 7 GiB of room against 103.7 GiB of files, so every refill after a prompt read the drive: prompts 15-40% slower than 0.1.34. - platform::file_cache_keeps(avail, arena, read): the decision, unit-tested in platform_memory_test with the reporter's numbers and the 32/64 GB PCs of #357/#362. - set_unbuffered: the arena is at most every expert's bytes, and the cache has to keep only the experts outside it. - recheck_unbuffered, after pin_cache_complement: the same decision from the RAM the copy really holds and the expert bytes outside it; switches either way (startup, nothing reading) and logs the numbers. - The 32 and 64 GB PCs still read unbuffered (the profile fill and the copy too, as before). Same tokens either way: only how the same bytes are read changes.

…ot keep it beside the RAM budget (Windows; Niko1221#286 ported into 0.1.31's tier) With --resident-budget-gib on a PC whose RAM cannot hold the expert files beside the budget, every expert outside the budget comes from the drive anyway. Through the mapped GGUF that costs twice: - the page faults are small requests on the critical path (the prompt path's stager, the profile fill); - the pages they bring in take the RAM the budget was sized for. On an emulated 32 GB PC, the 16 GiB budget was clamped to 11.65 GiB after the profile fill. - FileExpertSource::set_unbuffered: chosen by experts_unbuffered (Niko1221#357) with the budget as the RAM it keeps: whenever the files could not be kept beside it, cached now or not - a partly cached file's mapped pages would still land in the working set (seen on an emulated 64 GB PC: the budget shrank, then RAM ran out). Each shard is opened FILE_FLAG_NO_BUFFERING | OVERLAPPED. - copy_from_files, the decode prefetch and the complement copy read every role window of a batch at once, then copy it into place. - Windows that are less than 1 MiB apart merge into requests of up to 32 MiB. NTFS runs a file's unbuffered reads one at a time while the file is mapped anywhere, and the engine always maps the shards (token embedding, PLE). Measured on a PCIe 5 drive at queue depth 48: 512 KiB reads give 10.4 GB/s on an unmapped file and 3.4 GB/s on a mapped one; 8 MiB reads give 8.6 GB/s. - The profile fill reads its pairs in batches of 64, the next batch while this one is copied into the slots. - warms() is false when unbuffered. A PrefetchVirtualMemory of the mapping would read the experts through the cache, a second time. - Otherwise nothing changes: the mapped path stays for a keepable set, for no budget, and off Windows. STRATA_UNBUFFERED_LOAD=1/0 forces the choice. Greedy tokens are identical with the reads unbuffered, through the cache, and on 0.1.31.

…iko1221#452), Q5_0 experts on the GPU (Niko1221#473), IQ4_XS on AVX-2 (Niko1221#415), unbuffered loads on Windows (Niko1221#357 Niko1221#362), --peer-device (Niko1221#531), a 6 GB card starts (Niko1221#496), PR batch; setup requires it

…ain once the RAM copy is built (Niko1221#577) 0.1.38 made the choice before the resident set existed, from the budget asked for, against every shard's bytes (dense weights and the PLE table included). On a 96 GB PC with UD-Q4_K_XL (budget 72 GiB, 48.94 GiB really resident) that read 83.0 - 72 - 4 = 7 GiB of room against 103.7 GiB of files, so every refill after a prompt read the drive: prompts 15-40% slower than 0.1.34. - platform::file_cache_keeps(avail, arena, read): the decision, unit-tested in platform_memory_test with the reporter's numbers and the 32/64 GB PCs of Niko1221#357/Niko1221#362. - set_unbuffered: the arena is at most every expert's bytes, and the cache has to keep only the experts outside it. - recheck_unbuffered, after pin_cache_complement: the same decision from the RAM the copy really holds and the expert bytes outside it; switches either way (startup, nothing reading) and logs the numbers. - The 32 and 64 GB PCs still read unbuffered (the profile fill and the copy too, as before). Same tokens either way: only how the same bytes are read changes.

…ot keep it beside the RAM budget (Windows; Niko1221#286 ported into 0.1.31's tier) With --resident-budget-gib on a PC whose RAM cannot hold the expert files beside the budget, every expert outside the budget comes from the drive anyway. Through the mapped GGUF that costs twice: - the page faults are small requests on the critical path (the prompt path's stager, the profile fill); - the pages they bring in take the RAM the budget was sized for. On an emulated 32 GB PC, the 16 GiB budget was clamped to 11.65 GiB after the profile fill. - FileExpertSource::set_unbuffered: chosen by experts_unbuffered (Niko1221#357) with the budget as the RAM it keeps: whenever the files could not be kept beside it, cached now or not - a partly cached file's mapped pages would still land in the working set (seen on an emulated 64 GB PC: the budget shrank, then RAM ran out). Each shard is opened FILE_FLAG_NO_BUFFERING | OVERLAPPED. - copy_from_files, the decode prefetch and the complement copy read every role window of a batch at once, then copy it into place. - Windows that are less than 1 MiB apart merge into requests of up to 32 MiB. NTFS runs a file's unbuffered reads one at a time while the file is mapped anywhere, and the engine always maps the shards (token embedding, PLE). Measured on a PCIe 5 drive at queue depth 48: 512 KiB reads give 10.4 GB/s on an unmapped file and 3.4 GB/s on a mapped one; 8 MiB reads give 8.6 GB/s. - The profile fill reads its pairs in batches of 64, the next batch while this one is copied into the slots. - warms() is false when unbuffered. A PrefetchVirtualMemory of the mapping would read the experts through the cache, a second time. - Otherwise nothing changes: the mapped path stays for a keepable set, for no budget, and off Windows. STRATA_UNBUFFERED_LOAD=1/0 forces the choice. Greedy tokens are identical with the reads unbuffered, through the cache, and on 0.1.31. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…iko1221#452), Q5_0 experts on the GPU (Niko1221#473), IQ4_XS on AVX-2 (Niko1221#415), unbuffered loads on Windows (Niko1221#357 Niko1221#362), --peer-device (Niko1221#531), a 6 GB card starts (Niko1221#496), PR batch; setup requires it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…ain once the RAM copy is built (Niko1221#577) 0.1.38 made the choice before the resident set existed, from the budget asked for, against every shard's bytes (dense weights and the PLE table included). On a 96 GB PC with UD-Q4_K_XL (budget 72 GiB, 48.94 GiB really resident) that read 83.0 - 72 - 4 = 7 GiB of room against 103.7 GiB of files, so every refill after a prompt read the drive: prompts 15-40% slower than 0.1.34. - platform::file_cache_keeps(avail, arena, read): the decision, unit-tested in platform_memory_test with the reporter's numbers and the 32/64 GB PCs of Niko1221#357/Niko1221#362. - set_unbuffered: the arena is at most every expert's bytes, and the cache has to keep only the experts outside it. - recheck_unbuffered, after pin_cache_complement: the same decision from the RAM the copy really holds and the expert bytes outside it; switches either way (startup, nothing reading) and logs the numbers. - The 32 and 64 GB PCs still read unbuffered (the profile fill and the copy too, as before). Same tokens either way: only how the same bytes are read changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…early (Niko1221#285's part 3, kept here) Upstream got Niko1221#357/Niko1221#358 only: on IQ2_XS part 3 did not move the first decode step. With the NVFP4 pack it does: the 63 GiB arena overlaps the dense weights, the PLE table and the drafter, and the arena is loaded at 6.6-7.4 s instead of 8.0 (the same as 0.1.28-nvfp4.4's 6.7). The first cuBLAS handle is made on a thread meanwhile, so its VRAM is counted before the expert cache is sized - as in 0.1.28-nvfp4.4 (7,221 slots): the 1500 MiB reserve stays 1500. The card check now runs before the arena thread starts, so a build without code for the card stops at once. STRATA_ARENA_SYNC=1 keeps the old order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

sergiywith and others added 2 commits October 1, 2026 20:48

sergqwer force-pushed the pr/start-unbuffered branch from b26904d to 51feb9f Compare October 1, 2026 18:16

Niko1221 closed this Oct 3, 2026

This was referenced Oct 3, 2026

Thinking speed suddenly drops to 0.5 token/s #535

Closed

new error raise RuntimeError("the engine exited before it was ready" + (f" (see {log})" if log else "") + #517

Closed

sergqwer mentioned this pull request Oct 6, 2026

Start: register the expert arena per layer on a thread ahead of the readers (Windows; part 2 of #285) #1039

Closed

RichardFreml mentioned this pull request Oct 7, 2026

[Windows, RTX 5090 32 GB, NVFP4 fork] ple: --ple-io direct streams the n-gram table at ~34 MB/s while the SSD does 1.4 GB/s - fresh long prompts prefill ~5x slow; cold passes can trip the #29 watchdog #1425

Closed

Navigation Menu

Repository navigation

Start: read the expert arena unbuffered when the file cache cannot help (Windows; part 1 of #285) - #357

Start: read the expert arena unbuffered when the file cache cannot help (Windows; part 1 of #285)#357
sergqwer wants to merge 2 commits into
Niko1221:mainfrom
sergqwer:pr/start-unbuffered

sergqwer commented Oct 1, 2026

Uh oh!

sergqwer commented Oct 1, 2026

Uh oh!

sergqwer commented Oct 1, 2026

Uh oh!

Niko1221 commented Oct 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

sergqwer commented Oct 1, 2026

Start times

Uh oh!

sergqwer commented Oct 1, 2026

Uh oh!

sergqwer commented Oct 1, 2026

Uh oh!

Niko1221 commented Oct 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants