[JeanP00l]

What

STRATA_ARENA_MMAP=1 (opt-in, Linux; nothing changes without it) maps the host expert arena read-only from the pack's experts.bin instead of reading it into pinned, locked RAM. This is for machines whose GPUs already hold most experts and whose RAM is small. --mmap-experts refuses native packs (the Coder), so this path is for them.

Without it, the Coder (25 GB arena) on 2x 16 GB GPUs with 32 GB of RAM left ~1-3 GB available at 128K context. In long sessions the kernel swapped, and re-reading a multi-turn checkpoint took 25-80 s.

• First start: writes the arena to experts.bin (padded by one blob, so a whole-slot copy may start at any expert). Later starts map it.
• Page cache: keeps what the CPU pool and the prompt path read, and gives the rest back under pressure. A dropped page is re-read from the file, so no result depends on what is resident. The GPUs get no mapped alias: run with --pcie-frac 0.
• After the VRAM caches fill: pages of experts a GPU holds are released (MADV_DONTNEED; STRATA_ARENA_RELEASE=0 keeps them), and the rest are prefetched (MADV_WILLNEED). An adaptive swap releases the expert that went to VRAM and prefetches the one that came back.
• HIP: ROCclr copies a pageable source of >= 1 MiB by locking its pages in place, and keeps them locked. That undid every release. The engine raises GPU_PINNED_MIN_XFER_SIZE (an explicit setting wins). It locks an adaptive swap's source pages itself, only until the copy lands. The ranges are merged first, because a copy from a range only partly registered fails with invalid argument.
• STRATA_RSS_TRACE=1: prints RssFile at the startup steps. madvise reports success on locked pages and frees nothing, so RssFile is the honest check.
• Other platforms: Windows compiles it out. The upstream HIP compat gets cudaHostRegisterReadOnly.

Measured

2x AMD Instinct MI50 16 GB + 32 GB DDR4 (gfx906, see #638), Coder IQ1_M, 128K context, layer split:

	pinned arena	STRATA_ARENA_MMAP=1
RAM available while serving	~1 GB	25 GB
decode, ms per verify window	49.16	49.06

Status

Re-run on 2x MI50 with this branch on current main, results in the comment below.

🤖 Generated with Claude Code

[JeanP00l]

Re-run on the cards with this branch, merged with #638 and #639 (current main 99f3dbd). Same setup as the #639 runs: 2x MI50 16 GB, 32 GB DDR4, Coder IQ1_M, 131,072-token context, layer split 27, --pcie-frac 0. Each run is one engine start, a greedy 1,500-token code request and a 4-turn agent conversation.

	pinned arena (default)	STRATA_ARENA_MMAP=1
MemAvailable after start / after the requests	3.0 / 2.1 GiB	26.3 / 25.5 GiB (two runs)
decode, greedy 1,500 tokens	60.9 tok/s	61.0 tok/s
greedy text	same	same (bitwise)
agent turns (17K prompt, then 3 follow-ups)	47.4 / 9.0 / 9.7 / 12.8 s	46.8 / 9.2 / 9.9 / 13.0 s

A correction to the description. The "25-80 s" checkpoint re-reads came from long sessions on the port branch, once the machine ran out of RAM and swapped. This short 4-turn run does not reproduce them: with ~2 GiB available it has not reached that point yet. What it shows directly is RAM: about 23 GiB more is free with the same speed and the same text.

Marking ready.

[Niko1221]

Thanks, the ROCclr pin-in-place finding is useful beyond this PR (#579 and #590 also use --mmap-experts on HIP). This is in the 0.1.39 branch with two changes:

1. Spans gathered only when pinned: the adaptive swap loops called srcp->blob() for every swap to gather the spans pin_blobs locks, on every build and without the variable too. blob() counts reads, and with --mmap-experts the file tier may assemble the blob a second time. pin_blobs_on() now guards that, so CUDA and the runs without STRATA_ARENA_MMAP do exactly what 0.1.38 did.
2. Free-space check before the first write: the first mapped start writes the whole arena (25 GB for the Coder) next to the pack. It now checks the free space first (the arena plus 2 GiB). Without that room, it says so and keeps the arena in RAM for that start.

It builds for CUDA on Windows and Linux (the POSIX mapped arena compiles under Linux) and for Windows HIP. Default output is unchanged. We have no Linux AMD machine to run the mapped arena itself. Please post your re-run of this branch on the cards here; we'll keep the PR open until then and close it with the release.

[Niko1221]

Merged for 0.1.39 as STRATA_ARENA_MMAP=1 (Linux). We added a follow-up so experts.bin is written only when the drive has room.

Thanks @JeanP00l for the work and for correcting the description after the re-run.

[JeanP00l]

Quick post-merge check: the updated gfx906 candidate built all targets and started with STRATA_ARENA_MMAP=1 on MI50. It loaded the 23.42 GiB mapped expert arena from a read-only model mount and completed a deterministic inference whose generated code matched the existing gfx906 build. Available system memory remained about 25 GiB during the run and returned to about 28 GiB after teardown.

This confirms startup and inference through the mapped-arena path; it is not a new pinned-versus-mapped RAM or throughput A/B. The earlier controlled comparison remains the evidence for that benefit.