Start: read the expert arena unbuffered (the GGUF too, fixes #230), register it while it loads, on its own thread by sergqwer · Pull Request #285 · Niko1221/Strata

sergqwer · 2026-09-30T19:26:18Z

Summary

The expert arena's load was the start's long pole. It was a buffered read, then one CUDA registration of the whole
arena; registering 63 GiB of 4 KiB pages alone took 6.6 s. This PR changes four things:

Unbuffered reads.
- A pack's experts.bin is read with FILE_FLAG_NO_BUFFERING straight into the arena (16 readers, 8 MiB each).
- The GGUF branch too: this fixes The GGUF branch of load_experts_gguf still has the 4095-byte std::ifstream defect (0.02 -> 1.87 GiB/s, 42 min -> 54 s) #230. Setup's default native pack has no experts.bin, and its GGUF branch
  read through MSVC's std::ifstream in 4095-byte pieces: 0.02 GiB/s, 42 minutes in The GGUF branch of load_experts_gguf still has the 4095-byte std::ifstream defect (0.02 -> 1.87 GiB/s, 42 min -> 54 s) #230. Each chunk's 4 KiB-aligned
  window is now read unbuffered into an aligned buffer and scattered into the blobs. Off Windows the old reader
  stays.
Registration while loading. The arena is reserved first, and a thread registers its per-layer slices ahead of
the readers (PinnedArena::Deferred + register_slices). A reader waits for its slice: writing pages while
cudaHostRegister runs on them corrupted the arena under WDDM, even on large pages.
In parallel. The arena loads on its own thread while the main thread loads the dense weights, the PLE table,
the MTP drafter and the native head.
cuBLAS early. gemm_prewarm() creates the handle and runs its first GEMM off the critical path.

Measured

Measured on current main (0.1.29, d6708a4) with ISTA-DASLab's IQ2_XS, RTX 5090 32 GB, Ryzen 9 9950X3D, 128 GB DDR5, Windows 11. KL is first-token KL divergence (STRATA_DUMP_FIRST_LOGITS, #276); two runs of main itself differ by 0 on 5 of 6 short prompts and by 0.00013 on the sixth. IQ2_XS without experts.bin (setup's default pack), a 1K-token prompt and 64 tokens:

	arena read	start to exit
main (buffered, the file already in the OS cache)	33 GiB at 6.8 GiB/s	17.5 s
this PR (unbuffered from the NVMe)	33 GiB in 7.7 s, registration alongside	11.7 s

main had the advantage of the OS file cache here (128 GB of RAM). The 0.02 GiB/s of #230 appears on a cold start,
or when RAM is too small to cache the shards; the unbuffered reader does not depend on either. A pack with
experts.bin (fork, PCIe 5 drive, 63 GiB): 11.8 GiB/s instead of 3.3, first token after ~8 s instead of ~10.

Correctness

STRATA_VERIFY_ARENA=1 prints a checksum of the loaded arena. The unbuffered GGUF loader and the buffered one give
the same checksum (5217dac097a1751f), and the generated tokens are unchanged.

Switch

STRATA_BUFFERED_LOAD=1 keeps the old loaders.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…on its own thread The arena load was the start's long pole: a buffered read, then one CUDA registration of the whole arena (registering 63 GiB of 4 KiB pages alone took 6.6 s). - load_experts_direct (pinned.cu): a pack's experts.bin ranges are read with FILE_FLAG_NO_BUFFERING straight into the arena, 16 readers of 8 MiB. When a range is not 4 KiB-aligned it returns without an error and the buffered reader runs as before. - load_experts_gguf (the default native pack, no experts.bin): each chunk's 4 KiB-aligned window is read unbuffered into an aligned buffer and scattered into the blobs. MSVC's std::ifstream read it in 4095-byte pieces (#230: 0.02 GiB/s, 42 minutes); off Windows the old reader stays. - PinnedArena(bytes, bounds, Deferred) + register_slices(): the arena is reserved first and its per-layer slices are registered on a thread ahead of the readers; a reader waits for its slice (writing pages while cudaHostRegister runs on them corrupted the arena under WDDM, even on large pages). - generate.cpp: the arena loads on its own thread while the main thread loads the dense weights, the PLE table, the MTP drafter and the native head; it is joined before anything reads it. - gemm_prewarm(): the cuBLAS handle and its first GEMM are created early, off the critical path. IQ2_XS without experts.bin (setup's default): start to exit of a short generate 11.7 s vs 17.5 s, with main reading from a warm file cache; the arena checksum (STRATA_VERIFY_ARENA) is the same for both loaders. STRATA_BUFFERED_LOAD=1 keeps the old loaders (A/B). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

Niko1221 · 2026-10-01T06:25:00Z

Thanks - real work behind this. It conflicts with 0.1.31: the GGUF reader now opens a file per ROLE (Unsloth GGUFs split layers across shards; this PR opens one per layer), #230's FILE* reader landed, and the deferred register may undo #253's whole-arena pin on Linux. Please rebase and split into (1) the unbuffered reader, (2) the deferred/per-layer register (page-aligned bounds), (3) the arena thread + cuBLAS prewarm, each with cold/warm start times - note unbuffered reads skip the OS cache, so warm restarts (idle unload) must not get slower. Closing until then.

sergqwer · 2026-10-01T08:10:05Z

Measured before the split. IQ2_XS without experts.bin, start to exit of a 16-token generate, two runs each.

Cold means the IQ2_XS files were pushed out of the Windows file cache by reading 142 GB of other files first, so no admin is needed.

	arena loaded, warm	exit, warm	arena loaded, cold	exit, cold
0.1.31 main	4.2 s	7.4 s	20.0 s	24.5 s
this PR (on 0.1.29)	6.5 s	8.5 s	13.6-14.9 s	16.3-18.0 s

So #230's FILE* reader already makes 0.1.31 the faster one warm, as you expected for unbuffered reads. Unbuffered still saves ~7 s (30%) cold.

The plan for the split, on 0.1.31:

The unbuffered reader, Windows.
- It is used only when a quick buffered probe of the expert file says the file is not in the OS cache. A warm restart after an idle unload keeps 0.1.31's FILE* path, so it cannot get slower.
- Per-role files as expert_gguf_file() resolves them.
The per-layer registration ahead of the readers, Windows only.
- Within the same caps as now: STRATA_ARENA_PIN_GIB, the partial pin, the shared arena.
- Linux keeps [Linux][multi-GPU] 8 GiB cudaHostRegister cap causes ~3x Q2_0 prefill slowdown #253's whole-arena pin unchanged.
The arena thread and the cuBLAS prewarm.

Each will come with cold and warm times against 0.1.31.

🤖 Generated with Claude Code

sergqwer · 2026-10-01T09:25:41Z

Split and rebased on 0.1.31 as you asked:

Start: read the expert arena unbuffered when the file cache cannot help (Windows; part 1 of #285) #357, the unbuffered reader (Windows).
- It is used only when a cache probe finds the expert files uncached and the RAM beside the arena could not keep them for the next start.
- Otherwise The GGUF branch of load_experts_gguf still has the 4095-byte std::ifstream defect (0.02 -> 1.87 GiB/s, 42 min -> 54 s) #230's FILE* path runs as before. So a warm restart cannot get slower: 4.7 vs 4.6-5.1 s on 128 GB.
- On an emulated 64 GB PC the restart goes from 12.5-16.0 to 6.0-6.1 s.
- It resolves per-role files through expert_gguf_file().
Start: register the expert arena per layer on a thread ahead of the readers (Windows; part 2 of #285) #358, the per-layer registration ahead of the readers (Windows only, stacked on Start: read the expert arena unbuffered when the file cache cannot help (Windows; part 1 of #285) #357).
- Every reader waits for its layer's slice and the next one, so a page shared by two slices is never written mid-registration.
- Only where the whole arena would be registered anyway; Linux keeps [Linux][multi-GPU] 8 GiB cudaHostRegister cap causes ~3x Q2_0 prefill slowdown #253's whole-arena pin.
- With 4 KB pages the restart goes from 7.8-8.5 to 5.5-5.7 s, and cold from 12.4-13.0 to 11.1-11.5 s.
Part 3 (the arena thread + cuBLAS prewarm) is dropped. Measured on top of the other two, it finished the arena later (5.5 vs 4.5 s) and did not move the first decode step.

Each PR has cold and restart times against 0.1.31 at 128 GB and on an emulated 64 GB PC, with 4 KB and large pages.

🤖 Generated with Claude Code

…lp (Windows; part 1 of #285) A buffered read copies every expert through the file cache. That pays off only when the cache can keep the files for the next start. On a PC whose RAM cannot hold them beside the arena (a 64 GB PC with IQ2_XS: 24 GiB left for 36.5 GiB of files), every start reads the drive anyway, and the copy is pure cost. - experts_unbuffered() (pinned.cu): timed random 64 KiB reads tell whether the files are in the cache already (a cached read takes ~10 us, the drive ~80). Unbuffered is chosen only when they are not and the RAM left beside the arena could not keep them either. A warm restart that 0.1.31 serves from the cache stays on its FILE* path. STRATA_UNBUFFERED_LOAD=1/0 forces the choice; the log names the reader and why. - load_experts_gguf: each chunk's 4 KiB-aligned window is read with FILE_FLAG_NO_BUFFERING into an aligned buffer and scattered into the blobs, with a handle per role file as expert_gguf_file() resolves it (split shards). 16 readers. - load_experts_direct (pinned.cu): a pack's experts.bin ranges are read unbuffered straight into the arena. When the ranges are not 4 KiB-aligned, it falls back to load_experts_ranges. IQ2_XS without experts.bin, RTX 5090, emulated 64 GB PC (a locked-pages RAM ballast leaves 58 GiB), two rounds: | | first start: arena / exit | restart: arena / exit | |---|---|---| | 0.1.31 | 22.1-25.2 s / 28.2-31.0 s | 12.5-16.0 s / 17.5-20.8 s | | this | 18.3-20.2 s / 23.1-25.4 s | 6.0-6.1 s / 9.8-9.9 s | With 128 GB the probe finds the files cached (15 of 16 reads) and the buffered path runs as before. Greedy tokens are identical unbuffered, buffered and on 0.1.31 main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…eaders (Windows; part 2 of #285) 0.1.31 registers the whole arena with CUDA before the first byte is read. With 4 KB pages, which every account without the Lock Pages privilege gets, that registration is ~1.5 s of the start that nothing hides (IQ2_XS: 33 GiB). - PinnedArena(bytes, bounds, Deferred) reserves only. register_slices() registers the per-layer slices in order on a thread and publishes how many are done. - Every reader (load_experts_gguf buffered and unbuffered, load_experts_ranges, load_experts_direct) writes layer L only once slices L and L+1 are registered. Writing a page while cudaHostRegister runs on it corrupted the arena under WDDM, even on large pages. A layer boundary inside a page puts that page in both registrations, so the next slice must be done too (page-aligned in effect, without moving the slice bounds an expert must not straddle). - A slice CUDA refuses ends the registration, and the rest is locked resident as in the sliced fallback. - Only where the whole arena would be registered anyway: no cap (STRATA_ARENA_PIN_GIB, multi-GPU under WDDM) and no shared arena file. Linux keeps #253's whole-arena pin. STRATA_DEFERRED_REGISTER=0 restores the old order. - STRATA_VERIFY_ARENA=1 prints the loaded arena's FNV-1a checksum (tests). Checksums: the same with the old order and the deferred one, 4 KB and large pages, the GGUF in place and experts.bin (IQ2_XS; 13 deferred loads). IQ2_XS without experts.bin, RTX 5090, 128 GB. Each cell is the time until the arena is loaded / until a 16-token answer is done. Cold: the files pushed out of the file cache first. Two to three runs each: | | cold | restart | |---|---|---| | 4 KB pages (STRATA_NO_LARGEPAGES=1), 0.1.31 | 12.4-13.0 / 16.3-16.8 s | 7.8-8.5 / 10.9-11.7 s | | 4 KB pages, parts 1+2 | 11.1-11.5 / 14.9-15.4 s | 5.5-5.7 / 8.5-8.9 s | | large pages, 0.1.31 | 17.6-18.7 / 21.1-22.4 s | 4.6-5.1 / 7.0-7.6 s | | large pages, parts 1+2 | 17.6-19.9 / 21.3-23.6 s | 4.1-4.5 / 6.7-7.1 s | The cold spread with large pages comes from the drive: the same build gave 17.6 and 19.9 s with the deferred registration and the cache probe switched off as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

… layers 0.1.39's unbuffered reader (Niko1221#285 path for native packs) read only [lb, le) but reported the whole model's bytes, so a coordinator or worker refused its own correct load: 'ArenaExpertSource: the load read 25146163200 B of 16783769600'.

…eaders (Windows; part 2 of #285) 0.1.31 registers the whole arena with CUDA before the first byte is read. With 4 KB pages, which every account without the Lock Pages privilege gets, that registration is ~1.5 s of the start that nothing hides (IQ2_XS: 33 GiB). - PinnedArena(bytes, bounds, Deferred) reserves only. register_slices() registers the per-layer slices in order on a thread and publishes how many are done. - Every reader (load_experts_gguf buffered and unbuffered, load_experts_ranges, load_experts_direct) writes layer L only once slices L and L+1 are registered. Writing a page while cudaHostRegister runs on it corrupted the arena under WDDM, even on large pages. A layer boundary inside a page puts that page in both registrations, so the next slice must be done too (page-aligned in effect, without moving the slice bounds an expert must not straddle). - A slice CUDA refuses ends the registration, and the rest is locked resident as in the sliced fallback. - Only where the whole arena would be registered anyway: no cap (STRATA_ARENA_PIN_GIB, multi-GPU under WDDM) and no shared arena file. Linux keeps #253's whole-arena pin. STRATA_DEFERRED_REGISTER=0 restores the old order. - STRATA_VERIFY_ARENA=1 prints the loaded arena's FNV-1a checksum (tests). Checksums: the same with the old order and the deferred one, 4 KB and large pages, the GGUF in place and experts.bin (IQ2_XS; 13 deferred loads). IQ2_XS without experts.bin, RTX 5090, 128 GB. Each cell is the time until the arena is loaded / until a 16-token answer is done. Cold: the files pushed out of the file cache first. Two to three runs each: | | cold | restart | |---|---|---| | 4 KB pages (STRATA_NO_LARGEPAGES=1), 0.1.31 | 12.4-13.0 / 16.3-16.8 s | 7.8-8.5 / 10.9-11.7 s | | 4 KB pages, parts 1+2 | 11.1-11.5 / 14.9-15.4 s | 5.5-5.7 / 8.5-8.9 s | | large pages, 0.1.31 | 17.6-18.7 / 21.1-22.4 s | 4.6-5.1 / 7.0-7.6 s | | large pages, parts 1+2 | 17.6-19.9 / 21.3-23.6 s | 4.1-4.5 / 6.7-7.1 s | The cold spread with large pages comes from the drive: the same build gave 17.6 and 19.9 s with the deferred registration and the cache probe switched off as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…lp (Windows; part 1 of #285) A buffered read copies every expert through the file cache. That pays off only when the cache can keep the files for the next start. On a PC whose RAM cannot hold them beside the arena (a 64 GB PC with IQ2_XS: 24 GiB left for 36.5 GiB of files), every start reads the drive anyway, and the copy is pure cost. - experts_unbuffered() (pinned.cu): timed random 64 KiB reads tell whether the files are in the cache already (a cached read takes ~10 us, the drive ~80). Unbuffered is chosen only when they are not and the RAM left beside the arena could not keep them either. A warm restart that 0.1.31 serves from the cache stays on its FILE* path. STRATA_UNBUFFERED_LOAD=1/0 forces the choice; the log names the reader and why. - load_experts_gguf: each chunk's 4 KiB-aligned window is read with FILE_FLAG_NO_BUFFERING into an aligned buffer and scattered into the blobs, with a handle per role file as expert_gguf_file() resolves it (split shards). 16 readers. - load_experts_direct (pinned.cu): a pack's experts.bin ranges are read unbuffered straight into the arena. When the ranges are not 4 KiB-aligned, it falls back to load_experts_ranges. IQ2_XS without experts.bin, RTX 5090, emulated 64 GB PC (a locked-pages RAM ballast leaves 58 GiB), two rounds: | | first start: arena / exit | restart: arena / exit | |---|---|---| | 0.1.31 | 22.1-25.2 s / 28.2-31.0 s | 12.5-16.0 s / 17.5-20.8 s | | this | 18.3-20.2 s / 23.1-25.4 s | 6.0-6.1 s / 9.8-9.9 s | With 128 GB the probe finds the files cached (15 of 16 reads) and the buffered path runs as before. Greedy tokens are identical unbuffered, buffered and on 0.1.31 main.

…eaders (Windows; part 2 of #285) 0.1.31 registers the whole arena with CUDA before the first byte is read. With 4 KB pages, which every account without the Lock Pages privilege gets, that registration is ~1.5 s of the start that nothing hides (IQ2_XS: 33 GiB). - PinnedArena(bytes, bounds, Deferred) reserves only. register_slices() registers the per-layer slices in order on a thread and publishes how many are done. - Every reader (load_experts_gguf buffered and unbuffered, load_experts_ranges, load_experts_direct) writes layer L only once slices L and L+1 are registered. Writing a page while cudaHostRegister runs on it corrupted the arena under WDDM, even on large pages. A layer boundary inside a page puts that page in both registrations, so the next slice must be done too (page-aligned in effect, without moving the slice bounds an expert must not straddle). - A slice CUDA refuses ends the registration, and the rest is locked resident as in the sliced fallback. - Only where the whole arena would be registered anyway: no cap (STRATA_ARENA_PIN_GIB, multi-GPU under WDDM) and no shared arena file. Linux keeps #253's whole-arena pin. STRATA_DEFERRED_REGISTER=0 restores the old order. - STRATA_VERIFY_ARENA=1 prints the loaded arena's FNV-1a checksum (tests). Checksums: the same with the old order and the deferred one, 4 KB and large pages, the GGUF in place and experts.bin (IQ2_XS; 13 deferred loads). IQ2_XS without experts.bin, RTX 5090, 128 GB. Each cell is the time until the arena is loaded / until a 16-token answer is done. Cold: the files pushed out of the file cache first. Two to three runs each: | | cold | restart | |---|---|---| | 4 KB pages (STRATA_NO_LARGEPAGES=1), 0.1.31 | 12.4-13.0 / 16.3-16.8 s | 7.8-8.5 / 10.9-11.7 s | | 4 KB pages, parts 1+2 | 11.1-11.5 / 14.9-15.4 s | 5.5-5.7 / 8.5-8.9 s | | large pages, 0.1.31 | 17.6-18.7 / 21.1-22.4 s | 4.6-5.1 / 7.0-7.6 s | | large pages, parts 1+2 | 17.6-19.9 / 21.3-23.6 s | 4.1-4.5 / 6.7-7.1 s | The cold spread with large pages comes from the drive: the same build gave 17.6 and 19.9 s with the deferred registration and the cache probe switched off as well.

… layers 0.1.39's unbuffered reader (Niko1221#285 path for native packs) read only [lb, le) but reported the whole model's bytes, so a coordinator or worker refused its own correct load: 'ArenaExpertSource: the load read 25146163200 B of 16783769600'.

…lp (Windows; part 1 of Niko1221#285) A buffered read copies every expert through the file cache. That pays off only when the cache can keep the files for the next start. On a PC whose RAM cannot hold them beside the arena (a 64 GB PC with IQ2_XS: 24 GiB left for 36.5 GiB of files), every start reads the drive anyway, and the copy is pure cost. - experts_unbuffered() (pinned.cu): timed random 64 KiB reads tell whether the files are in the cache already (a cached read takes ~10 us, the drive ~80). Unbuffered is chosen only when they are not and the RAM left beside the arena could not keep them either. A warm restart that 0.1.31 serves from the cache stays on its FILE* path. STRATA_UNBUFFERED_LOAD=1/0 forces the choice; the log names the reader and why. - load_experts_gguf: each chunk's 4 KiB-aligned window is read with FILE_FLAG_NO_BUFFERING into an aligned buffer and scattered into the blobs, with a handle per role file as expert_gguf_file() resolves it (split shards). 16 readers. - load_experts_direct (pinned.cu): a pack's experts.bin ranges are read unbuffered straight into the arena. When the ranges are not 4 KiB-aligned, it falls back to load_experts_ranges. IQ2_XS without experts.bin, RTX 5090, emulated 64 GB PC (a locked-pages RAM ballast leaves 58 GiB), two rounds: | | first start: arena / exit | restart: arena / exit | |---|---|---| | 0.1.31 | 22.1-25.2 s / 28.2-31.0 s | 12.5-16.0 s / 17.5-20.8 s | | this | 18.3-20.2 s / 23.1-25.4 s | 6.0-6.1 s / 9.8-9.9 s | With 128 GB the probe finds the files cached (15 of 16 reads) and the buffered path runs as before. Greedy tokens are identical unbuffered, buffered and on 0.1.31 main.

…eaders (Windows; part 2 of Niko1221#285) 0.1.31 registers the whole arena with CUDA before the first byte is read. With 4 KB pages, which every account without the Lock Pages privilege gets, that registration is ~1.5 s of the start that nothing hides (IQ2_XS: 33 GiB). - PinnedArena(bytes, bounds, Deferred) reserves only. register_slices() registers the per-layer slices in order on a thread and publishes how many are done. - Every reader (load_experts_gguf buffered and unbuffered, load_experts_ranges, load_experts_direct) writes layer L only once slices L and L+1 are registered. Writing a page while cudaHostRegister runs on it corrupted the arena under WDDM, even on large pages. A layer boundary inside a page puts that page in both registrations, so the next slice must be done too (page-aligned in effect, without moving the slice bounds an expert must not straddle). - A slice CUDA refuses ends the registration, and the rest is locked resident as in the sliced fallback. - Only where the whole arena would be registered anyway: no cap (STRATA_ARENA_PIN_GIB, multi-GPU under WDDM) and no shared arena file. Linux keeps Niko1221#253's whole-arena pin. STRATA_DEFERRED_REGISTER=0 restores the old order. - STRATA_VERIFY_ARENA=1 prints the loaded arena's FNV-1a checksum (tests). Checksums: the same with the old order and the deferred one, 4 KB and large pages, the GGUF in place and experts.bin (IQ2_XS; 13 deferred loads). IQ2_XS without experts.bin, RTX 5090, 128 GB. Each cell is the time until the arena is loaded / until a 16-token answer is done. Cold: the files pushed out of the file cache first. Two to three runs each: | | cold | restart | |---|---|---| | 4 KB pages (STRATA_NO_LARGEPAGES=1), 0.1.31 | 12.4-13.0 / 16.3-16.8 s | 7.8-8.5 / 10.9-11.7 s | | 4 KB pages, parts 1+2 | 11.1-11.5 / 14.9-15.4 s | 5.5-5.7 / 8.5-8.9 s | | large pages, 0.1.31 | 17.6-18.7 / 21.1-22.4 s | 4.6-5.1 / 7.0-7.6 s | | large pages, parts 1+2 | 17.6-19.9 / 21.3-23.6 s | 4.1-4.5 / 6.7-7.1 s | The cold spread with large pages comes from the drive: the same build gave 17.6 and 19.9 s with the deferred registration and the cache probe switched off as well.

…lp (Windows; part 1 of Niko1221#285) A buffered read copies every expert through the file cache. That pays off only when the cache can keep the files for the next start. On a PC whose RAM cannot hold them beside the arena (a 64 GB PC with IQ2_XS: 24 GiB left for 36.5 GiB of files), every start reads the drive anyway, and the copy is pure cost. - experts_unbuffered() (pinned.cu): timed random 64 KiB reads tell whether the files are in the cache already (a cached read takes ~10 us, the drive ~80). Unbuffered is chosen only when they are not and the RAM left beside the arena could not keep them either. A warm restart that 0.1.31 serves from the cache stays on its FILE* path. STRATA_UNBUFFERED_LOAD=1/0 forces the choice; the log names the reader and why. - load_experts_gguf: each chunk's 4 KiB-aligned window is read with FILE_FLAG_NO_BUFFERING into an aligned buffer and scattered into the blobs, with a handle per role file as expert_gguf_file() resolves it (split shards). 16 readers. - load_experts_direct (pinned.cu): a pack's experts.bin ranges are read unbuffered straight into the arena. When the ranges are not 4 KiB-aligned, it falls back to load_experts_ranges. IQ2_XS without experts.bin, RTX 5090, emulated 64 GB PC (a locked-pages RAM ballast leaves 58 GiB), two rounds: | | first start: arena / exit | restart: arena / exit | |---|---|---| | 0.1.31 | 22.1-25.2 s / 28.2-31.0 s | 12.5-16.0 s / 17.5-20.8 s | | this | 18.3-20.2 s / 23.1-25.4 s | 6.0-6.1 s / 9.8-9.9 s | With 128 GB the probe finds the files cached (15 of 16 reads) and the buffered path runs as before. Greedy tokens are identical unbuffered, buffered and on 0.1.31 main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…eaders (Windows; part 2 of Niko1221#285) 0.1.31 registers the whole arena with CUDA before the first byte is read. With 4 KB pages, which every account without the Lock Pages privilege gets, that registration is ~1.5 s of the start that nothing hides (IQ2_XS: 33 GiB). - PinnedArena(bytes, bounds, Deferred) reserves only. register_slices() registers the per-layer slices in order on a thread and publishes how many are done. - Every reader (load_experts_gguf buffered and unbuffered, load_experts_ranges, load_experts_direct) writes layer L only once slices L and L+1 are registered. Writing a page while cudaHostRegister runs on it corrupted the arena under WDDM, even on large pages. A layer boundary inside a page puts that page in both registrations, so the next slice must be done too (page-aligned in effect, without moving the slice bounds an expert must not straddle). - A slice CUDA refuses ends the registration, and the rest is locked resident as in the sliced fallback. - Only where the whole arena would be registered anyway: no cap (STRATA_ARENA_PIN_GIB, multi-GPU under WDDM) and no shared arena file. Linux keeps Niko1221#253's whole-arena pin. STRATA_DEFERRED_REGISTER=0 restores the old order. - STRATA_VERIFY_ARENA=1 prints the loaded arena's FNV-1a checksum (tests). Checksums: the same with the old order and the deferred one, 4 KB and large pages, the GGUF in place and experts.bin (IQ2_XS; 13 deferred loads). IQ2_XS without experts.bin, RTX 5090, 128 GB. Each cell is the time until the arena is loaded / until a 16-token answer is done. Cold: the files pushed out of the file cache first. Two to three runs each: | | cold | restart | |---|---|---| | 4 KB pages (STRATA_NO_LARGEPAGES=1), 0.1.31 | 12.4-13.0 / 16.3-16.8 s | 7.8-8.5 / 10.9-11.7 s | | 4 KB pages, parts 1+2 | 11.1-11.5 / 14.9-15.4 s | 5.5-5.7 / 8.5-8.9 s | | large pages, 0.1.31 | 17.6-18.7 / 21.1-22.4 s | 4.6-5.1 / 7.0-7.6 s | | large pages, parts 1+2 | 17.6-19.9 / 21.3-23.6 s | 4.1-4.5 / 6.7-7.1 s | The cold spread with large pages comes from the drive: the same build gave 17.6 and 19.9 s with the deferred registration and the cache probe switched off as well.

…early (Niko1221#285's part 3, kept here) Upstream got Niko1221#357/Niko1221#358 only: on IQ2_XS part 3 did not move the first decode step. With the NVFP4 pack it does: the 63 GiB arena overlaps the dense weights, the PLE table and the drafter, and the arena is loaded at 6.6-7.4 s instead of 8.0 (the same as 0.1.28-nvfp4.4's 6.7). The first cuBLAS handle is made on a thread meanwhile, so its VRAM is counted before the expert cache is sized - as in 0.1.28-nvfp4.4 (7,221 slots): the 1500 MiB reserve stays 1500. The card check now runs before the arena thread starts, so a build without code for the card stops at once. STRATA_ARENA_SYNC=1 keeps the old order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…evice; requant checks the checkpoint's shapes - gemm_prewarm (Niko1221#285's part 3, kept in the fork) made a cuBLAS handle on the current device and gave it to the first Gemm, whatever its device. Under a layer split the stages' prompt paths start first, on other devices, and took a handle of device 0. The handle now goes only to a Gemm on the device it was made on. - tools/requant.py: a transposed checkpoint has the same byte count, so RTN or an all-Q8_0 plan could write a wrong-layout pack that passes the engine's size check; the layer's shapes are checked first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

… with one tiny product each (the first request -150 ms) The first FP16 and the first BF16 cublasGemmEx of a process load cuBLASLt's kernel libraries for that type (cuLibraryLoadData, ~65 ms each on an RTX 5090, 36 MiB of VRAM), whatever the shape, and the tray's first request after a load paid them inside its prompt. gemm_prewarm (Niko1221#285's thread, already making the cuBLAS handle while the files load) now runs one 16x256x256 product of each type into its own scratch (BF16 on sm_80+; not on HIP; not without a batched prompt path), and gemm_prewarm_wait lets the auto expert cache count those 36 MiB instead of the first prompt taking them from the reserve (so an auto cache has ~10 slots fewer). STRATA_GEMM_WARM=0 turns it off. Serve, the tray's args, 3 ABBA pairs: a first 2K request's TTFT 1110.5 -> 955.0 ms (-155 ms; the second request's level), the load time even (10.6 / 10.5 s), VRAM even; one-shot 2K TTFT 1252.7 -> 1100.7 ms (5 pairs). 32K even here: its first prompt waited for the PLE rows anyway. det_check and the 32K logits identical with a fixed cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

sergqwer mentioned this pull request Sep 30, 2026

Low RAM on Windows: a pinned tier of the most-read experts, the rest read unbuffered from experts.bin #286

Closed

Niko1221 closed this Oct 1, 2026

This was referenced Oct 1, 2026

Start: read the expert arena unbuffered when the file cache cannot help (Windows; part 1 of #285) #357

Closed

Start: register the expert arena per layer on a thread ahead of the readers (Windows; part 2 of #285) #358

Closed

sergqwer mentioned this pull request Oct 6, 2026

Start: register the expert arena per layer on a thread ahead of the readers (Windows; part 2 of #285) #1039

Closed

akivajp mentioned this pull request Oct 21, 2026

[Engine] Linux: read the experts from the GGUF with O_DIRECT under STRATA_UNBUFFERED_LOAD=1 #3624

Open

Navigation Menu

Repository navigation

Start: read the expert arena unbuffered (the GGUF too, fixes #230), register it while it loads, on its own thread - #285

Start: read the expert arena unbuffered (the GGUF too, fixes #230), register it while it loads, on its own thread#285
sergqwer wants to merge 1 commit into
Niko1221:mainfrom
sergqwer:pr/faster-start

sergqwer commented Sep 30, 2026

Uh oh!

Niko1221 commented Oct 1, 2026

Uh oh!

sergqwer commented Oct 1, 2026

Uh oh!

sergqwer commented Oct 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

sergqwer commented Sep 30, 2026

Summary

Measured

Correctness

Switch

Uh oh!

Niko1221 commented Oct 1, 2026

Uh oh!

sergqwer commented Oct 1, 2026

Uh oh!

sergqwer commented Oct 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants