Repository navigation
Feature Request: Make mmap vs read decision per tensor instead of per file. #27766
Description
Prerequisites
- I am running the latest code. Mention the version if possible as well.
- I carefully followed the README.md.
- I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- I reviewed the Discussions, and have a new and useful enhancement to share.
Feature Description
Currently running: build 10639 (5e6a37cb1).
I want to offload ngrams to disk, and stream them from NVMe. Currently trying to use --load-mode mmap --override-tensor per_layer_token_embd=CPU --ngl 99, but the load took 20+ minutes with no progress, and disk IO was absolutely blasted with reads.
I was measuring ~6200 major page faults per second. I read ~592Gib trying to load a 1003Gib model.
Motivation
Load ngrams from disk instead of RAM. Using a Strix Halo, which is UMA, RAM == VRAM, so loading them into RAM saves me nothing. I want to load them from disk instead. This would apply to any user with either a UMA machine, or lack of system RAM to offload into.
This also impacts any users using --override-tensor per_layer_token_embd=CPU on a model larger than their page cache.
This is specifically for unsloth/Qwen3.8-Flash-Next at UD-Q4_K_XL if that matters
Possible Implementation
Let me mmap just the ngram table.
Activity
jintakhan commentedon Aug 28, 2026
@danielhanchen I want to add on to this request. I am VRAM bound on an AMD dGPU and offloading MoE layers to system memory. --load-mode mmap currently offloads the ngram tables to SSD but destroys PP speed, while --load-mode none more than doubles PP speed at the cost of hosting the entire 51b ngram params in RAM. If we could mmap just the ngram table and offload it to SSD while keeping the rest of the model no mmap, we'd get the best of both worlds.
jintakhan commentedon Aug 30, 2026
Support for this has been merged, thanks @ngxson & co #27837
The commit makes the lazy-loading run independently of mmap setting and occur before the rest of the model loads.
--lazy-modeby default lazy loads only big chunks like the PLE tables, which stay on disk and streamed in as needed.This means PLE table SSD offload/mmap while direct memory loading of the rest of the model will become the default behavior of
--load-mode none. No other flags needed to achieve what we want. To load the full model to memory, PLE table and all,--load-mode none -lzm offwill restore the old--load-mode nonebehavior.Prefill speed gains peak at no-mmap speeds. Make sure model files are on low latency storage as latency now appears to be the bottleneck when streaming PLE tables from file.
Not the OP but the merge resolves the issue and issue is good to close.