[jesdga95]

On Linux every startup read went to the drive one ~128 KB request at a time (page faults through the mmap, or plain freads with the default readahead), so a cold start sat at queue depth ~2 for about fifteen minutes. Most of it was "filling the GPU's expert cache" at 30-39 MB/s from an NVMe that does ~3 GB/s with a deep queue. Windows already has an unbuffered, overlapped read path for this (set_unbuffered / read_direct); on Linux set_unbuffered returns false, so these reads went through the page cache one fault at a time.

This asks the kernel for the pages ahead of each read (madvise / posix_fadvise WILLNEED, in 128 KiB steps because WILLNEED only reads one readahead window per call) for the weights, the native dense matrices, the profile fill (256 pairs ahead), the peer tier's fill (--peer-device), the resident RAM copy, the MTP draft files, the native output head and token embedding, and strata-vision's projector. STRATA_READ_AHEAD=0 turns it all off, and each phase now logs how long it took.

Ryzen 9 9950X, RTX 5090, 32 GB DDR5, Gen3 NVMe, Ubuntu (kernel 7.0), v0.1.38, cold page cache:

• Q2_0 resident, 262K: ready in 70 s, was ~920 s. The fill went from ~39 MB/s to 3.2 GB/s (21.47 GiB in 7.1 s).
• IQ3_XXS resident + vision + peer tier (5060 Ti): ready in 32 s. MTP draft files 47 s → 0.3 s, peer fill 13.5 GiB in 5 s, 21.9 GiB RAM copy in 8 s, head/embedding/projector under a second each.
• Decode unchanged: struct still gives the same 7,230 tokens, and the bench is within noise.

Only tested on Linux with CUDA. On Windows the new helper is a no-op (the unbuffered path covers it), and I haven't compiled it on Windows or HIP. With a layer split, the first card's fill gets the read-ahead (q8atnight measured it below); the other cards' fills don't yet.

[q8atnight]

Carried this on a layer-split, low-RAM machine — it works there too, even though the PR
notes say "the layer-split fill isn't covered".

Setup: 2× RTX 5060 Ti 16 GB, layer split (--gpus 0,1), 32 GB RAM, low-RAM mode
(--mmap-experts, IQ2_XS native pack), Samsung PM981 Gen3 NVMe, Ubuntu 26.04, engine v0.1.39

• this PR, cold page cache (sync; echo 3 > drop_caches before each start):

	read-ahead off (STRATA_READ_AHEAD=0)	read-ahead on (default)
profile pre-fill of 8,581 slots on card 0	18.4 s (663 MB/s)	5.8 s (2,089 MB/s)
systemd start -> /health ok, cold cache	46 s	36 s

Decode unchanged, as you report. (The whole-box start gain is smaller than your Q2_0-resident
920 -> 70 s because in --mmap-experts mode far less is read at startup — but the expert
pre-fill itself still gains ~3x on the layer-split path.)

No conflicts carrying it on v0.1.39 except one trivial block in memory.hpp where #577's
file_cache_keeps() and the new advise_willneed() declarations touch the same lines —
both kept, builds clean (CUDA 13.4, sm_120).

[jesdga95]

Thanks for running it on a split setup, that's a case I couldn't test. You're right: the first card's fill goes through the same loop, so it gets the read-ahead; only the other cards' fills (the per-stage loop) don't. I've fixed the description.

Also rebased onto v0.1.39. Same memory.hpp resolution as yours, both declarations kept.

[jesdga95]

Numbers for the peer fill that went into the last push (--peer-device 1, RTX 5060 Ti 16 GB as the peer, IQ3_XXS resident at 262K, same box, cold page cache):

• before: the peer fill read at ~12 MB/s (queue depth ~2) and was on track for 15+ minutes
• after: peer GPU 1: 8270 experts, 13.51 GiB (filled in 31.6 s), ~460 MB/s

That's still well below the main fill's 3.2 GB/s through the same read-ahead, so something else in that loop (probably the async copy to the second card from pageable memory) is the next limit. I haven't dug into it.

[jesdga95]

Pushed one more commit (5949e9d): the same read-ahead for the last few slow reads at startup, plus timing lines for them. With the peer tier on, these were still reading one page fault at a time and made up most of the untimed part of the start:

• native output head (437 MB, cudaMemcpy from the mapping): now 0.1 s
• token embedding (260 MiB, memcpy from the mapping): now 0.1 s
• strata-vision's projector (907 MB through llama.cpp's loader, before the engine even starts): now 0.7 s, warm-up 0.5 s. It honours STRATA_READ_AHEAD=0 too.

Same box, IQ3_XXS resident + vision + --peer-device 1, cold page cache: ready in 32 s, was 120 s. What's left is mostly the drive at full speed (GPU fill 7.1 s, peer fill 5.0 s, RAM copy 8 s) plus ~5 s pinning the 21.9 GiB complement.

[blange48]

Feedback (rig: 4 x RTX 5080 16 GB (PCIe Gen3, one card at x8), Linux, IQ3_S, 262K context, int8 K/V with --kv-resident 32768, layer split 12,24,36 with --trim-stage-weights, "parallel"/--batch 8 --batch-groups 4, MTP spec 4, 123 GB RAM, NVMe): applied on top of 0.1.39 and in production since today.

• Good: no regression. The engine starts and the exactness checks (batch_test.py with 8 and 3 slots, parking_test.py) stay identical.
• Not shown here: our restarts are warm. The pod goes from created to ready in 90 s, with ~60 GB of the model already in the page cache, so this doesn't exercise the cold path your numbers are about. We haven't measured a real cold start (after dropping the caches) yet.

[Niko1221]

Merged in 0.1.40: read-ahead at startup on Linux (STRATA_READ_AHEAD=0 turns it off). The cold start numbers and the split-machine run are in the release notes.