Low RAM on Windows: a pinned tier of the most-read experts, the rest read unbuffered from experts.bin by sergqwer · Pull Request #286 · Niko1221/Strata

sergqwer · 2026-09-30T19:26:20Z

Summary

Stacked on #285 (it uses its deferred per-layer registration). Only the last commit is new here.

The resident arena holds every expert, including the VRAM cache's share of them a second time. A PC whose RAM cannot
hold them beside the system has setup's low-RAM mode, --mmap-experts, which leaves the experts to the OS file
cache. On Windows that cache fills RAM and gets trimmed, and decode falls off. On the NVFP4 fork, copying blobs
through the mapped file made Windows trim and decode fell from 72 to 33 tok/s.

TieredExpertSource keeps host copies only where the engine reads them:

Pinned tier. A pinned, registered tier (large pages where allowed) holds as many experts as fit in the free RAM
minus STRATA_RAM_RESERVE_GIB (default 6). They go in by rank: first the experts outside VRAM, then the VRAM
cache's tail that the prompt path borrows. The GPU copies them by DMA like the full arena's.
Everything else is read from experts.bin unbuffered, so the file cache neither grows nor gets trimmed. A
decode token's misses are prefetched per layer (begin_layer), the prompt path's stager reads them with
read_blob, and lent slots are refilled from the file.
The adaptive tier stays in step. An expert leaving VRAM without a host copy gets a tier slot (copied back from
its VRAM slot first); one entering VRAM gives its slot back.

--low-ram, --no-low-ram and --ram-budget GIB control it. It also switches on by itself when the pack has
experts.bin (setup writes it only for its low-RAM mode) and the experts plus 16 GiB exceed the RAM installed. It
needs one GPU, an --expert-profile, and no remote experts. setup's low-RAM mode passes --low-ram on Windows
instead of --mmap-experts. The loader is Windows-only for now, so Linux keeps --mmap-experts.

Measured

Measured on current main (0.1.29, d6708a4) with ISTA-DASLab's IQ2_XS, RTX 5090 32 GB, Ryzen 9 9950X3D, 128 GB DDR5, Windows 11. KL is first-token KL divergence (STRATA_DUMP_FIRST_LOGITS, #276); two runs of main itself differ by 0 on 5 of 6 short prompts and by 0.00013 on the sixth.

Correct to the token. With the same expert cache (17,806 slots), no PCIe share and a static cache, the full
arena and the tier (--ram-budget 8) generate the same 128 tokens. STRATA_TIER_VERIFY compared 5,460 host
copies with the file, and later 5,604 including the ones copied back from VRAM by the adaptive tier: 0 differ.
An 8 GiB tier on IQ2_XS (1,338 of the 6,800 experts outside VRAM come from the file): 137.7 tok/s decode vs
149.6 with the full arena (static cache, no PCIe share, 128 tokens).
The NVFP4 fork (63 GiB of experts, RAM emulated by locking 60 of 128 GB):

64 GB of RAM with	decode	32K prompt, first token
RTX 5090 (32 GB)	112-118 tok/s (same as with 96+ GB)	9.3 s (96+ GB: 5.7 s)
a 24 GB card	86-90 tok/s (96+ GB: 95)
a 16 GB card	54-56 tok/s (96+ GB: 67)

With 96 GB or more nothing changes: first-token KL to the arena is exactly 0.

Switch

--no-low-ram gives the full arena; --mmap-experts still selects the mapped file.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…on its own thread The arena load was the start's long pole: a buffered read, then one CUDA registration of the whole arena (registering 63 GiB of 4 KiB pages alone took 6.6 s). - load_experts_direct (pinned.cu): a pack's experts.bin ranges are read with FILE_FLAG_NO_BUFFERING straight into the arena, 16 readers of 8 MiB. When a range is not 4 KiB-aligned it returns without an error and the buffered reader runs as before. - load_experts_gguf (the default native pack, no experts.bin): each chunk's 4 KiB-aligned window is read unbuffered into an aligned buffer and scattered into the blobs. MSVC's std::ifstream read it in 4095-byte pieces (#230: 0.02 GiB/s, 42 minutes); off Windows the old reader stays. - PinnedArena(bytes, bounds, Deferred) + register_slices(): the arena is reserved first and its per-layer slices are registered on a thread ahead of the readers; a reader waits for its slice (writing pages while cudaHostRegister runs on them corrupted the arena under WDDM, even on large pages). - generate.cpp: the arena loads on its own thread while the main thread loads the dense weights, the PLE table, the MTP drafter and the native head; it is joined before anything reads it. - gemm_prewarm(): the cuBLAS handle and its first GEMM are created early, off the critical path. IQ2_XS without experts.bin (setup's default): start to exit of a short generate 11.7 s vs 17.5 s, with main reading from a warm file cache; the arena checksum (STRATA_VERIFY_ARENA) is the same for both loaders. STRATA_BUFFERED_LOAD=1 keeps the old loaders (A/B). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…rest from experts.bin (Windows) The resident arena holds every expert - with the VRAM cache's share of them a second time - so a PC whose RAM cannot hold them beside the system cannot run it, and --mmap-experts leaves them to the OS file cache. TieredExpertSource keeps host copies only where the engine reads them: a pinned, registered tier (large pages when allowed) sized to the free RAM minus STRATA_RAM_RESERVE_GIB (default 6), filled with the experts outside VRAM by rank, then the VRAM cache's tail that the prompt path borrows. Everything else is read from experts.bin unbuffered: a decode token's misses are prefetched at begin_layer, the prompt path's stager reads them with read_blob, and lent slots are refilled from the file. The adaptive tier keeps it in step: an expert leaving VRAM without a host copy gets a tier slot (copied back from its VRAM slot first), one entering VRAM gives its slot back. - --low-ram / --no-low-ram / --ram-budget GIB; by itself when the pack has experts.bin and the experts plus 16 GiB exceed the RAM installed (one GPU, an --expert-profile, no remote experts). - setup.py: its low-RAM mode passes --low-ram on Windows instead of --mmap-experts. - dispatcher: the PCIe share only for experts with a device alias (dma_capable). - STRATA_TIER_VERIFY / STRATA_TIER_STATS for tests. IQ2_XS: the same 128 tokens as the full arena with the same cache; every host copy equals the file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

Niko1221 · 2026-10-01T06:25:05Z

Thanks! 0.1.31 added a tier for this case: --mmap-experts reads a native pack's experts from the GGUF in place, --resident-budget-gib N keeps the hottest in RAM, plus parallel fetch and routing prefetch. Please benchmark this against that on the same machine; whatever wins can go into the existing tier. Closing until then.

sergqwer · 2026-10-01T08:10:04Z

Benchmarked as you asked, on the same machine: our tier against 0.1.31's --resident-budget-gib.

Setup:

IQ2_XS (33 GiB of experts).
An emulated 32 GB PC with a 16 GB GPU: a locked-pages RAM ballast leaves 26 GiB available, and --vram-reserve-mib 16900.
Both tiers with a 16 GiB RAM budget.
The 8K prompt, then a 256-token answer.
Two runs each, alternated.

	decode	time to first token	whole run
0.1.31 `--resident-budget-gib 16`, GGUF in place	70.3 / 73.8 tok/s	23.4 / 26.2 s	42.7 / 45.9 s
this PR `--low-ram --ram-budget 16`, experts.bin	91.1 / 90.8 tok/s	5.7 / 6.1 s	18.0 / 19.3 s

What the logs show:

Prompt path, 0.1.31: "experts streamed 21012, host 9,244-11,199 ms" - the mapped reads land on the critical path.
Prompt path, ours: "host 2,987-2,996 ms" - the misses are read unbuffered into pinned slots, ahead of the layer.
Decode file reads, 0.1.31: 6.4-7.3 GB per answer (9-12 ms/round of reading); the routing prefetch had warmed 8-16% of them.
0.1.31 with the experts.bin pack (mapped): prefill was the same (732 tok/s). Available RAM then fell below 1 GB, and my guard stopped both runs. The file cache takes what the budget leaves.

Since ours wins here, I'll port it into the existing 0.1.31 tier rather than keep a second tier, and resubmit on top of the #285 split:

the pinned budget;
unbuffered prefetch into tier slots at begin_layer;
the prompt stager's read_blob;
the VRAM <-> tier exchange.

🤖 Generated with Claude Code

sergqwer · 2026-10-01T09:55:40Z

Ported into 0.1.31's tier as #362 (stacked on #357), so there is no second tier.

What moved over: the unbuffered reads of everything outside the RAM budget. They now read the GGUF in place per role, in overlapped batches.

What did not need porting:

The pinned budget: your complement is page-locked already. It had only shrunk to 11.5-13.6 GiB because the mapped profile fill ate the RAM first.
The VRAM <-> RAM exchange: already there.

Results on the same emulated 32 GB PC + 16 GB GPU, 16 GiB budget, same day:

Decode: 84-89 vs 67-69 tok/s.
8K prompt + 256 tokens, done at 21.5-22.1 vs 45.1-47.3 s from launch.
The budget holds its full 16 GiB.

One finding that shapes the rest: NTFS serializes unbuffered reads of a file that is mapped anywhere.

512 KiB reads drop from 10.4 to 3.4 GB/s.
The engine maps the shards for the embedding and PLE.
So File tier: read the GGUF in place unbuffered when the file cache cannot keep it beside the RAM budget (Windows; #286 ported into the existing tier) #362 merges nearby windows into large requests. Details and the remaining gap to the old experts.bin tier are in the PR.

🤖 Generated with Claude Code

…ot keep it beside the RAM budget (Windows; #286 ported into 0.1.31's tier) With --resident-budget-gib on a PC whose RAM cannot hold the expert files beside the budget, every expert outside the budget comes from the drive anyway. Through the mapped GGUF that costs twice: - the page faults are small requests on the critical path (the prompt path's stager, the profile fill); - the pages they bring in take the RAM the budget was sized for. On an emulated 32 GB PC, the 16 GiB budget was clamped to 11.65 GiB after the profile fill. - FileExpertSource::set_unbuffered: chosen by experts_unbuffered (#357) with the budget as the RAM it keeps: whenever the files could not be kept beside it, cached now or not - a partly cached file's mapped pages would still land in the working set (seen on an emulated 64 GB PC: the budget shrank, then RAM ran out). Each shard is opened FILE_FLAG_NO_BUFFERING | OVERLAPPED. - copy_from_files, the decode prefetch and the complement copy read every role window of a batch at once, then copy it into place. - Windows that are less than 1 MiB apart merge into requests of up to 32 MiB. NTFS runs a file's unbuffered reads one at a time while the file is mapped anywhere, and the engine always maps the shards (token embedding, PLE). Measured on a PCIe 5 drive at queue depth 48: 512 KiB reads give 10.4 GB/s on an unmapped file and 3.4 GB/s on a mapped one; 8 MiB reads give 8.6 GB/s. - The profile fill reads its pairs in batches of 64, the next batch while this one is copied into the slots. - warms() is false when unbuffered. A PrefetchVirtualMemory of the mapping would read the experts through the cache, a second time. - Otherwise nothing changes: the mapped path stays for a keepable set, for no budget, and off Windows. STRATA_UNBUFFERED_LOAD=1/0 forces the choice. Greedy tokens are identical with the reads unbuffered, through the cache, and on 0.1.31. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

setup's low-RAM packs have an experts.bin, and with it 0.1.31's tier mapped it: on an emulated 32 GB PC the mapped reads took the RAM the budget was sized for, and available RAM fell below 1 GB (#286, measured yesterday). - set_unbuffered also takes experts.bin (paths_ is the one file); read_direct reads one window per blob (a blob is contiguous there) and merges near ones as for the GGUF. - blob() hands out the stage buffers for any unbuffered source (staged()), so prefetch, begin_layer and transient work as with the GGUF in place; the mapped experts.bin path is unchanged otherwise. IQ2_XS pack with experts.bin, emulated 32 GB PC + 16 GB GPU, 16 GiB budget, 8K prompt + 256 tokens, two runs: decode 100.2 / 100.6 tok/s, done at 17.2 / 15.5 s, the prompt path's host copies 2.1 s (the GGUF in place: 84-89 tok/s, 21.5-22.1 s, 4.3 s). Greedy tokens identical unbuffered, through the cache and on 0.1.31. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…ot keep it beside the RAM budget (Windows; #286 ported into 0.1.31's tier) With --resident-budget-gib on a PC whose RAM cannot hold the expert files beside the budget, every expert outside the budget comes from the drive anyway. Through the mapped GGUF that costs twice: - the page faults are small requests on the critical path (the prompt path's stager, the profile fill); - the pages they bring in take the RAM the budget was sized for. On an emulated 32 GB PC, the 16 GiB budget was clamped to 11.65 GiB after the profile fill. - FileExpertSource::set_unbuffered: chosen by experts_unbuffered (#357) with the budget as the RAM it keeps: whenever the files could not be kept beside it, cached now or not - a partly cached file's mapped pages would still land in the working set (seen on an emulated 64 GB PC: the budget shrank, then RAM ran out). Each shard is opened FILE_FLAG_NO_BUFFERING | OVERLAPPED. - copy_from_files, the decode prefetch and the complement copy read every role window of a batch at once, then copy it into place. - Windows that are less than 1 MiB apart merge into requests of up to 32 MiB. NTFS runs a file's unbuffered reads one at a time while the file is mapped anywhere, and the engine always maps the shards (token embedding, PLE). Measured on a PCIe 5 drive at queue depth 48: 512 KiB reads give 10.4 GB/s on an unmapped file and 3.4 GB/s on a mapped one; 8 MiB reads give 8.6 GB/s. - The profile fill reads its pairs in batches of 64, the next batch while this one is copied into the slots. - warms() is false when unbuffered. A PrefetchVirtualMemory of the mapping would read the experts through the cache, a second time. - Otherwise nothing changes: the mapped path stays for a keepable set, for no budget, and off Windows. STRATA_UNBUFFERED_LOAD=1/0 forces the choice. Greedy tokens are identical with the reads unbuffered, through the cache, and on 0.1.31. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

setup's low-RAM packs have an experts.bin, and with it 0.1.31's tier mapped it: on an emulated 32 GB PC the mapped reads took the RAM the budget was sized for, and available RAM fell below 1 GB (#286, measured yesterday). - set_unbuffered also takes experts.bin (paths_ is the one file); read_direct reads one window per blob (a blob is contiguous there) and merges near ones as for the GGUF. - blob() hands out the stage buffers for any unbuffered source (staged()), so prefetch, begin_layer and transient work as with the GGUF in place; the mapped experts.bin path is unchanged otherwise. IQ2_XS pack with experts.bin, emulated 32 GB PC + 16 GB GPU, 16 GiB budget, 8K prompt + 256 tokens, two runs: decode 100.2 / 100.6 tok/s, done at 17.2 / 15.5 s, the prompt path's host copies 2.1 s (the GGUF in place: 84-89 tok/s, 21.5-22.1 s, 4.3 s). Greedy tokens identical unbuffered, through the cache and on 0.1.31. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…nchronous reads) Niko1221#286's unbuffered file tier existed on Windows alone: on Linux the experts outside the RAM copy were always read through the page cache, as page faults of the mapped GGUF - small requests on the critical path, and pages that a 32 GB PC cannot keep beside a 16 GiB RAM copy anyway, so they were read again and again. - open_direct opens the files with O_DIRECT; read_direct queues every 4 KiB-aligned window of a batch with one io_submit (linux/aio_abi.h, no library; allowed by Docker's default seccomp profile, unlike io_uring) and waits for all of them, each thread with its own context and aligned buffer. Windows merges windows up to 1 MiB apart for NTFS; here only windows that touch are merged. A refused submit or a failed read falls back to the mapping, as on Windows. - The same decision as Windows' `cache_counts = false` arm: unbuffered when the file cache could not keep the expert bytes outside the RAM copy (platform::file_cache_keeps, from MemAvailable and the tightest cgroup limit), re-checked once the copy is built (Niko1221#577). STRATA_UNBUFFERED_LOAD=1 / 0 forces it either way. A pack's experts.bin works too (its Linux open now records the path). - direct_fallbacks() counts the blobs an unbuffered read could not deliver. - Tests: file_expert_source_test reads a synthetic pack unbuffered - one blob at a time, a batch of adjacent experts, copy_blob - and requires the file's bytes and no fallback (a mutation that fails every read is caught). ctest: 63 of 66 pass; the three others need a Q2_0 pack, a higher `ulimit -l` and AVX-512, as before. Measured on an RTX 3080 Ti Laptop 16 GB, i9-12900H, 30 GB RAM, KIOXIA PCIe 4 NVMe, Linux 7.0, ext4: Qwen3.8-Flash-Next Abliterated Q4_K_M in place (--native), --resident-budget-gib 16, --expert-cache auto, 8K context, greedy; three prompts (code, Russian prose, a 7.8K-token summary), each with and without MTP, two rounds of buffered (STRATA_UNBUFFERED_LOAD=0) against unbuffered, the same binary. Decode 4.38 -> 7.90 tok/s on average (x1.80; x1.49-x2.01 per prompt and mode), the 7.8K prompt read at 154/172 -> 184/290 tok/s, startup 36 -> 18-21 s. The code prompt's answer is the same text in all 16 runs; the other two differ between runs of either arm (the adaptive GPU cache changes which experts the CPU computes), as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ide Niko1221#286's unbuffered Windows reads, the CPU tier off when a peer GPU prefills - gemm: upstream's FP16 path below sm_80 replaces the local BF16->FP16 conversion (same effect on sm_70) - expert_source: the RAM copy keeps the per-role passes and read_blob (Linux pread); the unbuffered batches (Niko1221#286) take whole blobs in one pass when direct_ is open - generate: the profile fill keeps upstream's inline fill with prefetch_pairs when the file tier is unbuffered, otherwise the threaded fill_slots - prefill: the CPU tier's rows follow the GPU's (bounds = gpu_rows); it is off with --peer-device (m.pp); the next stage's thread drops the session affinity (run_impl, next_run_) - server: the grammar (response_format) import beside the think-literal helpers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…es the keep set Upstream (v0.1.40, expert_source.cpp file_tier_unbuffered and platform::file_cache_keeps, Niko1221#286 / Niko1221#577 / Niko1221#773): the hottest experts get a RAM copy (our RAM tier); the rest are read through the file cache, or with O_DIRECT when the cache could not keep them (available RAM less 4 GiB below their bytes). It never drops pages after a buffered read (its notes: POSIX_FADV_DONTNEED made decode re-read from the disk). This replaces 7bd95ad's keep set and per-expert DONTNEED. The O_DIRECT reader covers each expert with its 4 KiB-aligned window. expert_stream_test runs both modes on a fake pack whose experts do not start on 4 KiB boundaries: 600 of 600 slots correct each (RTX 3060). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…ot keep it beside the RAM budget (Windows; #286 ported into 0.1.31's tier) With --resident-budget-gib on a PC whose RAM cannot hold the expert files beside the budget, every expert outside the budget comes from the drive anyway. Through the mapped GGUF that costs twice: - the page faults are small requests on the critical path (the prompt path's stager, the profile fill); - the pages they bring in take the RAM the budget was sized for. On an emulated 32 GB PC, the 16 GiB budget was clamped to 11.65 GiB after the profile fill. - FileExpertSource::set_unbuffered: chosen by experts_unbuffered (#357) with the budget as the RAM it keeps: whenever the files could not be kept beside it, cached now or not - a partly cached file's mapped pages would still land in the working set (seen on an emulated 64 GB PC: the budget shrank, then RAM ran out). Each shard is opened FILE_FLAG_NO_BUFFERING | OVERLAPPED. - copy_from_files, the decode prefetch and the complement copy read every role window of a batch at once, then copy it into place. - Windows that are less than 1 MiB apart merge into requests of up to 32 MiB. NTFS runs a file's unbuffered reads one at a time while the file is mapped anywhere, and the engine always maps the shards (token embedding, PLE). Measured on a PCIe 5 drive at queue depth 48: 512 KiB reads give 10.4 GB/s on an unmapped file and 3.4 GB/s on a mapped one; 8 MiB reads give 8.6 GB/s. - The profile fill reads its pairs in batches of 64, the next batch while this one is copied into the slots. - warms() is false when unbuffered. A PrefetchVirtualMemory of the mapping would read the experts through the cache, a second time. - Otherwise nothing changes: the mapped path stays for a keepable set, for no budget, and off Windows. STRATA_UNBUFFERED_LOAD=1/0 forces the choice. Greedy tokens are identical with the reads unbuffered, through the cache, and on 0.1.31.

setup's low-RAM packs have an experts.bin, and with it 0.1.31's tier mapped it: on an emulated 32 GB PC the mapped reads took the RAM the budget was sized for, and available RAM fell below 1 GB (#286, measured yesterday). - set_unbuffered also takes experts.bin (paths_ is the one file); read_direct reads one window per blob (a blob is contiguous there) and merges near ones as for the GGUF. - blob() hands out the stage buffers for any unbuffered source (staged()), so prefetch, begin_layer and transient work as with the GGUF in place; the mapped experts.bin path is unchanged otherwise. IQ2_XS pack with experts.bin, emulated 32 GB PC + 16 GB GPU, 16 GiB budget, 8K prompt + 256 tokens, two runs: decode 100.2 / 100.6 tok/s, done at 17.2 / 15.5 s, the prompt path's host copies 2.1 s (the GGUF in place: 84-89 tok/s, 21.5-22.1 s, 4.3 s). Greedy tokens identical unbuffered, through the cache and on 0.1.31.

…ot keep it beside the RAM budget (Windows; Niko1221#286 ported into 0.1.31's tier) With --resident-budget-gib on a PC whose RAM cannot hold the expert files beside the budget, every expert outside the budget comes from the drive anyway. Through the mapped GGUF that costs twice: - the page faults are small requests on the critical path (the prompt path's stager, the profile fill); - the pages they bring in take the RAM the budget was sized for. On an emulated 32 GB PC, the 16 GiB budget was clamped to 11.65 GiB after the profile fill. - FileExpertSource::set_unbuffered: chosen by experts_unbuffered (Niko1221#357) with the budget as the RAM it keeps: whenever the files could not be kept beside it, cached now or not - a partly cached file's mapped pages would still land in the working set (seen on an emulated 64 GB PC: the budget shrank, then RAM ran out). Each shard is opened FILE_FLAG_NO_BUFFERING | OVERLAPPED. - copy_from_files, the decode prefetch and the complement copy read every role window of a batch at once, then copy it into place. - Windows that are less than 1 MiB apart merge into requests of up to 32 MiB. NTFS runs a file's unbuffered reads one at a time while the file is mapped anywhere, and the engine always maps the shards (token embedding, PLE). Measured on a PCIe 5 drive at queue depth 48: 512 KiB reads give 10.4 GB/s on an unmapped file and 3.4 GB/s on a mapped one; 8 MiB reads give 8.6 GB/s. - The profile fill reads its pairs in batches of 64, the next batch while this one is copied into the slots. - warms() is false when unbuffered. A PrefetchVirtualMemory of the mapping would read the experts through the cache, a second time. - Otherwise nothing changes: the mapped path stays for a keepable set, for no budget, and off Windows. STRATA_UNBUFFERED_LOAD=1/0 forces the choice. Greedy tokens are identical with the reads unbuffered, through the cache, and on 0.1.31.

setup's low-RAM packs have an experts.bin, and with it 0.1.31's tier mapped it: on an emulated 32 GB PC the mapped reads took the RAM the budget was sized for, and available RAM fell below 1 GB (Niko1221#286, measured yesterday). - set_unbuffered also takes experts.bin (paths_ is the one file); read_direct reads one window per blob (a blob is contiguous there) and merges near ones as for the GGUF. - blob() hands out the stage buffers for any unbuffered source (staged()), so prefetch, begin_layer and transient work as with the GGUF in place; the mapped experts.bin path is unchanged otherwise. IQ2_XS pack with experts.bin, emulated 32 GB PC + 16 GB GPU, 16 GiB budget, 8K prompt + 256 tokens, two runs: decode 100.2 / 100.6 tok/s, done at 17.2 / 15.5 s, the prompt path's host copies 2.1 s (the GGUF in place: 84-89 tok/s, 21.5-22.1 s, 4.3 s). Greedy tokens identical unbuffered, through the cache and on 0.1.31.

…ot keep it beside the RAM budget (Windows; Niko1221#286 ported into 0.1.31's tier) With --resident-budget-gib on a PC whose RAM cannot hold the expert files beside the budget, every expert outside the budget comes from the drive anyway. Through the mapped GGUF that costs twice: - the page faults are small requests on the critical path (the prompt path's stager, the profile fill); - the pages they bring in take the RAM the budget was sized for. On an emulated 32 GB PC, the 16 GiB budget was clamped to 11.65 GiB after the profile fill. - FileExpertSource::set_unbuffered: chosen by experts_unbuffered (Niko1221#357) with the budget as the RAM it keeps: whenever the files could not be kept beside it, cached now or not - a partly cached file's mapped pages would still land in the working set (seen on an emulated 64 GB PC: the budget shrank, then RAM ran out). Each shard is opened FILE_FLAG_NO_BUFFERING | OVERLAPPED. - copy_from_files, the decode prefetch and the complement copy read every role window of a batch at once, then copy it into place. - Windows that are less than 1 MiB apart merge into requests of up to 32 MiB. NTFS runs a file's unbuffered reads one at a time while the file is mapped anywhere, and the engine always maps the shards (token embedding, PLE). Measured on a PCIe 5 drive at queue depth 48: 512 KiB reads give 10.4 GB/s on an unmapped file and 3.4 GB/s on a mapped one; 8 MiB reads give 8.6 GB/s. - The profile fill reads its pairs in batches of 64, the next batch while this one is copied into the slots. - warms() is false when unbuffered. A PrefetchVirtualMemory of the mapping would read the experts through the cache, a second time. - Otherwise nothing changes: the mapped path stays for a keepable set, for no budget, and off Windows. STRATA_UNBUFFERED_LOAD=1/0 forces the choice. Greedy tokens are identical with the reads unbuffered, through the cache, and on 0.1.31. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

setup's low-RAM packs have an experts.bin, and with it 0.1.31's tier mapped it: on an emulated 32 GB PC the mapped reads took the RAM the budget was sized for, and available RAM fell below 1 GB (Niko1221#286, measured yesterday). - set_unbuffered also takes experts.bin (paths_ is the one file); read_direct reads one window per blob (a blob is contiguous there) and merges near ones as for the GGUF. - blob() hands out the stage buffers for any unbuffered source (staged()), so prefetch, begin_layer and transient work as with the GGUF in place; the mapped experts.bin path is unchanged otherwise. IQ2_XS pack with experts.bin, emulated 32 GB PC + 16 GB GPU, 16 GiB budget, 8K prompt + 256 tokens, two runs: decode 100.2 / 100.6 tok/s, done at 17.2 / 15.5 s, the prompt path's host copies 2.1 s (the GGUF in place: 84-89 tok/s, 21.5-22.1 s, 4.3 s). Greedy tokens identical unbuffered, through the cache and on 0.1.31. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

Call-scope ResidentExecutor over ResidentSlots: lease-bound submit, committed-progress observation with separate resource/output watermarks, idempotent cancel/drain, generation-fenced synchronous certified snapshot borrow/return. Focused CPU contract regressions. No model issuance, production call-site migration, allocation reduction, real-request qualification, latency/speedup or promotion; Niko1221#286 remains open for the transaction-scoped migration.

sergiywith and others added 2 commits September 30, 2026 22:25

Niko1221 closed this Oct 1, 2026

sergqwer mentioned this pull request Oct 1, 2026

File tier: read the GGUF in place unbuffered when the file cache cannot keep it beside the RAM budget (Windows; #286 ported into the existing tier) #362

Closed

icocoon mentioned this pull request Oct 2, 2026

Windows + 32 GB RAM: the low-RAM resident mode never engages - the fit check runs after the engine's own cache warm-up #467

Closed

jesdga95 mentioned this pull request Oct 3, 2026

Linux: read ahead at startup (cold start ~920 s → 70 s) #699

Closed

cakescats mentioned this pull request Oct 4, 2026

file tier: unbuffered reads on Linux too (O_DIRECT + the kernel's asynchronous reads) #773

Closed

willthehuman mentioned this pull request Oct 6, 2026

file_cache_keeps() picks unbuffered (O_DIRECT) expert reads on a 32 GB + 12 GB GPU box with --resident-budget-gib, halving decode; STRATA_UNBUFFERED_LOAD=0 restores it #1194

Closed

sergqwer mentioned this pull request Oct 9, 2026

Resident RAM mode on Windows: the start-up fill and the RAM copy read experts.bin unbuffered (ready 42 -> 10 s with 21 GiB free) #1696

Closed

Navigation Menu

Repository navigation

Low RAM on Windows: a pinned tier of the most-read experts, the rest read unbuffered from experts.bin - #286

Low RAM on Windows: a pinned tier of the most-read experts, the rest read unbuffered from experts.bin#286
sergqwer wants to merge 2 commits into
Niko1221:mainfrom
sergqwer:pr/low-ram-tier

sergqwer commented Sep 30, 2026

Uh oh!

Niko1221 commented Oct 1, 2026

Uh oh!

sergqwer commented Oct 1, 2026

Uh oh!

sergqwer commented Oct 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

sergqwer commented Sep 30, 2026

Summary

Measured

Switch

Uh oh!

Niko1221 commented Oct 1, 2026

Uh oh!

sergqwer commented Oct 1, 2026

Uh oh!

sergqwer commented Oct 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants