Resident RAM mode on a layer split — 3 GPUs, UD-IQ4_XS, 40 GB RAM (engine 0.1.40) #1145

Resident RAM mode on a layer split — 3 GPUs, UD-IQ4_XS, 40 GB RAM (engine 0.1.40)

Answering the "30-minute soak of the resident RAM mode on a split (#848)" call. Note this is 3 GPUs, one of them over USB4 — beyond the 2-GPU case you measured.

Rig

• 7700X running Ubuntu 24.04.5 LTS (GNU/Linux 7.0.14-15-pve x86_64)
• RTX 5070 Ti (PCIe5 x8, probe 28.9 GB/s) + RTX 5060 Ti (PCIe4 x8, 14.4 GB/s) + RTX 5060 Ti on USB4 (3.8 GB/s) = 48GB VRAM
• LXC container: 40 GB RAM, models on local NVMe
• UD-IQ4_XS, --layer-split 30,40, --kv k8v4, --max-context 204800, --resident-experts, conversation cache 12 GiB / 16 slots
• Workload: 3 agent conversations, 110–120K tokens each, alternating turns

A/B: full complement (31.58 GiB) vs --resident-budget-gib 18

Reading: with the full complement, prompt streaming misses the pinned complement and reads the GGUF at NVMe speed (436 tok/s × 3.48 MB/blob ≈ 4.5 GB/s), and the 31.58 GiB page-lock starves the conversation cache. Budget 18 GiB (8,056 experts by profile rank) covers the prompt hot set: prefill 3.1× faster and parking works again, at ~30 % decode (adaptive swaps evict outside the complement — files … blobs grows ~30 k reads per request). For reference, IQ3_S on this rig in plain mmap mode (0.1.39) did 1,268 tok/s at 126K — UD-IQ4_XS with budget 18 matches it despite 31 % larger blobs.

Two observations

1. 7438 verified additional-GPU experts remain on the mmap fallback — the CUDA1/2 cache contents are not in the resident complement, so adaptive swaps involving them read files. Should the complement (or the budget) cover them?
2. the file tier reads unbuffered … the file cache cannot keep them — with budget 18 there are 18.5 GiB free, yet the whole 37.4 GiB is read O_DIRECT. Would a budgeted run gain from letting the page cache hold the overflow just past the budget, instead of the file?

⚠️ No crashes, NaN windows or stalls in 2h of interactive use so far; soak continues.

setup.py: the #498 guard is stale for 0.1.40. split_budget() strips --resident-budget-gib on any multi-GPU start and demands download_gb + 24 ≈ 118 GB for UD-IQ4_XS, pointing 40 GB rigs at "start it on one GPU". But with 0.1.40 (#848) the budget runs on a layer split — verified above, this whole test is that config. The strip path seems to need the same engine-version gate split_mmap() already uses (RESIDENT_SPLIT_ENGINE). My config was hand-built from the IQ3_S one because setup would not offer it; it works, and the budget is what makes 40 GB viable.