Repository navigation
Feature Request: Two-tier GPU+RAM expert cache for MoE offload (pluggable eviction policy) #20757
Description
Prerequisites
- I am running the latest code.
- I carefully followed the README.md.
- I searched for existing issues and found none covering a persistent VRAM expert slot cache.
- I reviewed the Discussions, and have a new and useful enhancement to share.
Feature Description
Problem
Running GPT-OSS-120B (128 experts/layer, 36 layers, ~57 GB expert weights) on an 8 GB GPU works today via --cpu-moe with MoE tensors kept in CPU RAM. The dense/attention layers run on GPU; expert weights are copied CPU→GPU on each forward pass by ggml_backend_sched_compute_splits().
The bottleneck: every decode step copies the same ~4 hot experts per layer from RAM to GPU, uses them, and discards the GPU copy. For a 36-layer model with top-4 routing that's 144 RAM→GPU copies per token — most of them identical to the previous token. The GPU-side expert data is thrown away between passes even though the same experts will be needed again immediately.
MoE routing is skewed. In practice ~15–20% of experts handle ~80% of tokens. Keeping those hot experts resident in VRAM between passes — in a fixed-address slot buffer — reduces per-token PCIe traffic to near zero after a short warm-up period.
Proof of concept
I built a Python PoC (tinyserve) on HuggingFace transformers for GPT-OSS-120B implementing a two-tier expert cache:
Tier 1: GPU VRAM — persistent slot buffer, N fixed-address expert slots, SLRU eviction [proposed]
Tier 2: CPU RAM — pinned memory, backing store for all local experts [proposed]
Tier 3: SSD/mmap — full weight tensor, demand-paged by OS (existing)
On a cache hit (Tier 1), the expert is already on GPU — zero copies. On a Tier 1 miss, the expert is copied from pinned RAM (Tier 2) into the freed slot. On a Tier 2 miss (cold start), the expert is demand-paged from the mmap'd file.
Results on RTX PRO 2000 (8 GB VRAM), GPT-OSS-120B:
| Phase | tok/s | Cache hit rate |
|---|---|---|
| Cold start (0–80 t) | 1.9–2.5 | 48–56% |
| Warming (80–160 t) | 2.6–5.1 | 53–78% |
| Steady state (160+) | 12–14 | ~98–100% |
Pure CPU offload with no cache on the same hardware: ~0.5–1 tok/s. At steady state the model is GPU-compute bound — the PCIe pipe is mostly idle.
What llama.cpp already has
-
--cpu-moeand--n-cpu-moein all tools (common/arg.cpp:2284,2291) — already route MoE expert tensors to CPU RAM for all tools includingllama-cli,llama-server, andllama-bench. Also available as env varsLLAMA_ARG_CPU_MOE/LLAMA_ARG_N_CPU_MOE. This part is done; the gap is what happens after the weights are on CPU. -
Selective expert copy in
ggml_backend_sched_compute_splits()(ggml/src/ggml-backend.cpp:1445–1564) — already reads theidstensor at runtime and copies only the used expert sub-rows CPU→GPU viaggml_backend_tensor_set_async(). Crucially, it copies by byte offset into a GPU tensor that mirrors the full CPU tensor layout (expert_offset = first_id * expert_size, line 1529) — no slot remapping, no persistence across passes. This is the hook point for the cache. -
POSIX_MADV_WILLNEEDinllama_mmap(src/llama-mmap.cpp:436) — used at model load time for prefetch.POSIX_MADV_DONTNEEDdoes not currently exist in the codebase and would need to be added for page release.
What is new
Tier 1: persistent GPU slot buffer
Today the GPU-side expert allocation (input_cpy in ggml_backend_sched_compute_splits()) mirrors the full CPU tensor shape — it has n_expert slots even though only top_k are used. Each forward pass, the selected experts are written into their original offset positions, then discarded at pass end. Expert IDs sent to the kernel are global IDs that index this full-size tensor.
The proposed change: allocate a GPU buffer of N slots at model load (N << n_expert, controlled by --moe-expert-cache-size N). A persistent mapping (expert_id → slot_idx) carries over between passes. On a hit, the slot is already populated — zero copy. On a miss, a slot is evicted per policy, the new expert is copied from Tier 2 into that slot, and the mapping is updated. The kernel receives remapped slot indices rather than global expert IDs, requiring a small change to the GGML_OP_MUL_MAT_ID path in ggml_backend_sched_compute_splits().
This is the primary performance lever. The 14 tok/s in the PoC comes from Tier 1 hits.
Tier 2: pinned RAM backing store
Today, CPU-side expert data lives in the mmap'd weight file — demand-paged from SSD on first access, held in OS page cache thereafter. No pinned allocation is made.
The proposed change: at model load, allocate a CPU pinned memory region for all local experts. Hot experts are available for fast non-blocking H2D transfer on Tier 1 misses. Without pinned memory, every Tier 1 miss copies from pageable memory, which is slower and cannot overlap with GPU compute.
Proposed design
1. Pluggable eviction policy interface
To maximize flexibility and allow further advancements without touching cache logic, the policy is plug-and-play. Swapping policies requires no cache code changes:
// Sketch — naming should follow ggml conventions
struct ggml_moe_cache_policy {
const char * name;
void (* on_hit) (void * ctx, int32_t expert_id, int32_t slot);
int32_t (* select_evict)(void * ctx);
void (* on_load) (void * ctx, int32_t expert_id, int32_t slot);
void * ctx;
};
// Selected via --moe-expert-cache-policy <name>
void ggml_moe_cache_register_policy(struct ggml_moe_cache_policy policy);
// Built-in policies
struct ggml_moe_cache_policy ggml_moe_cache_policy_lru (int capacity);
struct ggml_moe_cache_policy ggml_moe_cache_policy_slru(int capacity);
struct ggml_moe_cache_policy ggml_moe_cache_policy_lfu (int capacity);
struct ggml_moe_cache_policy ggml_moe_cache_policy_fifo(int capacity);2. SLRU as the recommended default
SLRU is the recommended default for the Tier 1 slot buffer. The reason is specific to how MoE routing interacts with the two-tier design.
Plain LRU has a failure mode here: prefill sequences activate all experts in a roughly flat distribution, wiping the GPU slot buffer. When decode resumes, Tier 1 is cold — every token falls through to a Tier 2 copy (CPU→GPU) until the cache re-warms. In a two-tier system this is particularly costly because even Tier 2 hits involve a PCIe copy, eliminating the primary performance advantage.
SLRU splits the slot buffer into a probationary segment (~20% of slots) for new admissions and a protected segment (~80%) for experts accessed more than once. Eviction always targets probationary first. A prefill burst fills and cycles through the probationary segment but cannot displace protected entries. Hot decode experts, having been seen multiple times, occupy protected slots and survive the burst intact.
The PoC also uses a frequency-gated admission filter: an expert is only admitted to the Tier 1 slot buffer on its second miss. Cold experts seen once during a prefill never enter either segment, protecting both from pollution at the point of admission rather than only at eviction. In the PoC, SLRU + admission filter improved steady-state hit rate by 8–15 percentage points over plain LRU on mixed prefill+decode traffic.
3. SSD-tier management — PoC tricks (~60 LOC)
Two optimisations from the PoC transfer directly:
- Readahead (
POSIX_MADV_WILLNEED): after each decode step, fire a prefetch hint for the experts just used — they're likely needed next token. The pattern exists atsrc/llama-mmap.cpp:436for model load; the same call applied per-expert after each pass costs nothing and gives the OS time to warm the pages. - Page release (
POSIX_MADV_DONTNEED): when an expert is evicted from Tier 2, release its OS page cache entry. This call does not currently exist in the codebase and would need to be added tosrc/llama-mmap.cpp.
A further optimisation from the PoC — double-buffering the PCIe transfer (two alternating staging buffers that overlap Tier 2→Tier 1 copies with GPU compute) — is out of scope for a first implementation but is a natural follow-on once the slot buffer is in place.
Help wanted
I'm primarily a Python developer. The PoC demonstrates the approach works and the numbers are real, but I'm not equipped to write production-quality C++ for the ggml backend scheduler internals.
Looking for:
- A C++ contributor familiar with
ggml_backend_sched_compute_splits()to implement the slot buffer and two-tier cache - Feedback on whether the policy interface fits ggml conventions — specifically whether the slot-index remapping belongs inside
ggml_backend_sched_compute_splits()or as a new op/wrapper - Any existing work in this direction I might have missed
Happy to provide benchmark data, answer design questions, and test any implementation.
Files relevant to a C++ implementer
| File | Relevance |
|---|---|
ggml/src/ggml-backend.cpp:1445–1564 |
Expert copy loop — both tiers insert here; slot remap needed at line 1529 |
common/arg.cpp:2284,2291 |
Existing --cpu-moe / --n-cpu-moe parameter wiring (already in all tools) |
src/llama.cpp:481–509 |
MoE tensor CPU placement logic for auto-fit |
src/llama-mmap.cpp:436 |
POSIX_MADV_WILLNEED pattern; DONTNEED needs adding |
AI disclosure: I used Claude Code to help analyse the llama.cpp codebase and structure this RFC. The Python PoC, benchmark numbers, and design rationale are my own. I reviewed this text myself and can explain and defend every technical claim here.
Activity
lee-b commentedon Mar 31, 2026
I think a simpler approach might be possible here: use imatrix data to decide which experts are "hot" enough to justify pinning into VRAM. Let the other expert "slots" churn. Many GGUF repo's already include the imatrix files, of course, so the optimisation could be applied automatically even, in many cases, or specified with a path to an imatrix file.
AI sums it up like this:
Data Source
The "hotness" info comes from imatrix files (GGUF format). The imatrix tool collects:
Simplified Approach (vs the full multi-tier cache)
Instead of implementing the complex SLRU cache from the issue, here's a much simpler approach:
At model load: Use imatrix to decide which experts go where
At runtime: Let existing scheduler handle the rest
So most tokens should require zero expert copies, if they use the most frequent experts. A few tokens would be slower, depending on how much offload is done.
I think this would also get a lot of the benefits of the multi-tier approach just by letting the system (and the user / sysadmin) manage which experts are in system RAM vs. (priority-sorted) swap vs. nvme vs. SSD, or whatever.
@ggerganov , thoughts?
zoepatterson1231-del commentedon Mar 31, 2026
martinalderson commentedon Apr 7, 2026
I've been working on a proof-of-concept implementation of this in the ggml backend (Vulkan, AMD RX 9070 XT 16GB) with Gemma 4 26B-A4B (128 experts, 8 selected, 30 layers).
Approach:
input_cpyis impossible)cpy_tensor_async(batched in compute command buffer)set_tensor_async+ GPU→GPU update to cache frominput_cpyGGML_OP_OFFLOAD_MIN_BATCH=1to force MUL_MAT_ID to GPU during decode (default scheduler keeps MoE on CPU for batch_size=1)ggml_vk_buffer_write_2d_asyncbypassing staging buffers (see Feature Request: Vulkan: bypass staging buffer when destination is host-visible (SAM/ReBAR) #21590)Results (all experts on CPU,
-ngl 30 -ot exps=CPU):Cache hit rate climbs from 47% at 100 tokens to 67% at 500+ tokens. Perplexity verified identical within error bars (8662 ± 1173 vs 8861 ± 1200 on test corpus).
Key finding: the existing
set_tensor_asyncfor non-pinned (mmap'd) data is fully synchronous in the Vulkan backend — each call goes through staging buffer +ggml_vk_synchronize(). This is the main bottleneck. The SAM bypass eliminates this. Without SAM, forcing GPU offload for MoE actually makes decode slower (9 tok/s vs 28) because the per-expert sync overhead dominates.I also built an MoE activation profiler (
examples/moe-profile/) and a Blazor WASM visualisation tool (tools/moe-viz/) with animated heatmaps, per-domain comparison, and decode stats across 25 bundled profiles — that's what led me to the caching approach. The profiling confirmed decode routing is heavily skewed (Gini ≈ 0.76) with 44% expert overlap between consecutive tokens.Screenshot from the tool if anyone is interested:
Code is on my fork's
moe-profilebranch (fair warning — it's messy with experiments, not PR-ready): https://github.com/martinalderson/llama.cpp/tree/moe-profile.Btw I'm new here and not super familiar with the codebase, this is very much Claude Code helping me out learn and experiment, hope it's useful.
Interpause commentedon Apr 8, 2026
hm why did the issue get closed? But I don't see who closed it is github glitching?
viewerbeware789 commentedon Apr 14, 2026
Hi. I'm new to github. Where can I check to see if this was implemented? I see that the issue is closed but how can I find out if it was implemented?
Merge moe-expert-cache-v2: MoE expert-weight cache for the CUDA backend
kisasexypantera94 commentedon May 15, 2026
I ran a small side experiment on the mmap / file-backed part of MoE expert residency, mostly with Apple / unified-memory machines in mind.
This is not an end-to-end runtime benchmark. It is just:
preadmicrobench on the generated rangesSetup
Qwen3-30B-A3B-Q4_K_M.gguf,Qwen3-30B-A3B-Q6_K.gguf(layer, expert_id)hit_avg/p10/minare aggregated across runsI also tried frozen counters and sliding-window counters. Both were worse than cumulative on this workload, so the numbers below are cumulative-only.
GGUF layout
For both quants, the expert layout is convenient for file-backed loading. Each
(layer, expert)is 3 contiguous file ranges:Extracted layout:
So at least for these GGUFs, this is not a strided-read problem. Loading one expert means reading 3 large contiguous ranges.
Results
The
preadnumbers are not cold-SSD numbers. I did not useF_NOCACHE, so macOS page cache is involved. I mostly used this to check the actual range pattern and syscall/read overhead, not raw SSD latency.Coalescing / overread
I also checked whether it is worth reading extra bytes to merge nearby ranges.
For Q4_K_M, budget 64, refresh every 8 tokens:
So I would not start with aggressive overread. Sorting ranges by offset and coalescing only adjacent ranges looks like a reasonable first version.
Runtime angle
For unified-memory systems, the runtime shape I want to try is:
The split that matters:
initial/prewarm: one-time startup costrefresh: background work, ideally hidden behind computeLFU-cold miss: blocking path, so the miss payload has to stay smallI’m planning to look into a runtime integration for the Apple/unified-memory + mmap-backed residency path first, rather than trying to solve the full cross-backend cache design immediately. Happy to share the tracing/layout/read-plan scripts if useful.
kisasexypantera94 commentedon May 17, 2026
Quick update on Apple / unified-memory / mmap.
I built a runtime MoE expert pool: hot experts live in a compact Metal slot pool, routing IDs are remapped through slot_table, and observer/LFU is fed from selected_experts. The mechanics work: bit-exact full/identity paths and ~97-99% steady-state hit rate, matching offline sim.
Memory-wise, the first result was negative. With POOL_LAYERS=4 N_SLOTS=96, the pool adds ~1.3 GiB footprint because the original mmap-backed tensors are still loaded and reachable.
The important finding is macOS accounting:
-nglbarely reduces it: Q4_K_M full GPU ~23.5 GiB wired, ngl=24 ~21.7, ngl=12 ~20.7. Butngl=0drops it hard to ~2.3 GiB, so the big wired cost comes from involving Metal at all.MADV_DONTNEEDon the mmap region tanks throughput while Metal buffers are alive;mprotect(PROT_NONE)does not affect compute.The path that still makes sense is to keep cold expert tensors out of Metal at load time: store them only as metadata, then
preadinto the compact pool when needed. I’ll try that by adding a selective loader path forffn_*_exps.Implement MoE expert residency PoC
kisasexypantera94 commentedon May 17, 2026
Following up on my previous comment.
Branch:
kisasexypantera94/llama.cpp:moe-expert-residencyHow it works
Expert tensors are loaded with
--no-mmap --override-tensor =CPU, so they live in anonymous CPU pages. Separately, each MoE layer gets a compact Metal pool withN_SLOTSexpert slots, kept outside the scheduler buffers.A small
ggml_custom_4dCPU op sits on the router path. It readsselected_experts, checks whether each expert is already in the Metal pool, and remaps expert IDs to slot IDs. On miss, it picks a slot with LRU,preads the expert weights from the GGUF file, uploads them into the pool withggml_backend_tensor_set, and writes the remapped slot ID.After that, the normal
MUL_MAT_IDpath runs unchanged, just against the compact pool instead of the full expert tensor.The original CPU pages are released with
MADV_FREEafter first use.Results
Qwen3-30B-A3B-Q6_K, Apple M3 Pro 36 GB, 200 generated tokens:
Generation quality looks correct in all tested configs.
Notes
The
N_SLOTS=128row is the important one. There is effectively no eviction there, but generation still drops from 40.4 to 20.3 t/s. So the main cost is not LRU orpread; it is the CPU custom op creating a cross-backend sync point on every MoE layer.On Apple Silicon this should be avoidable, since CPU and GPU share memory. The remap path probably needs to move out of a CPU ggml op and closer to the Metal side.
Prompt processing is bad in all configs because misses are handled synchronously.
The code is still rough: env vars, inline GGUF parsing, debug logging, etc. I’m cleaning it up incrementally and will try to turn it into a PR once the design is less hacky.
cli/moe: good defaults — predictive prefetch on by default; flags tun…
MoE expert offloading: research + activation-aware cache, CUDA SLRU p…
batot1 commentedon Jun 12, 2026
Results
I measured the decode behavior of
GGML_OP_OFFLOAD_MIN_BATCH=1on a partially CPU-resident MoE model. The environment variable works as intended, but forcing decode-time offload without persistent expert residency causes a large TG regression.Setup
Decode performance
GGML_OP_OFFLOAD_MIN_BATCH=1Expert split placement
GGML_OP_OFFLOAD_MIN_BATCH=1GGML_OP_OFFLOAD_MIN_BATCH=1Decode-time H2D traffic
GGML_OP_OFFLOAD_MIN_BATCH=1GGML_OP_OFFLOAD_MIN_BATCH=1Interpretation
32is justified for decode;batch_size=1decode offload without persistent expert residency is counterproductive;I am currently measuring inter-token expert locality and simulated LRU hit rates for 8, 16, 32 and 64 persistent expert slots per tensor.
These results also seem relevant to #18530 and #18535.
Lidenburg commentedon Jun 14, 2026
For anyone who, like me, has kept this issue opened in a tab and kept track on the comments:
I've basically implemented the initial 3 tier expert cache in a new fork at https://github.com/Lidenburg/llama.cpp
I would be curious to see what speeds people get out of it on their hardware. Especially when using disk offloading on a new gen fast nvme SSD, it wasn't great on my ~500mb/s SATA SSD that i tried it with, but I'm pretty sure with a new gen one with 7, 10 or 15 gb/s one it could be very usable.
Someone also opened a new discussion about it at #24528
JigSawPT commentedon Jul 4, 2026
I built a working prototype of this on top of
--n-cpu-moeand measured it end-to-end on consumer hardware (RTX 5090 32 GB + 64 GB DDR5-6000), so sharing in case it is useful input for this feature.It is a static, load-time hot/cold split — not a dynamic eviction cache. At model load it keeps a VRAM copy of the top-N most-activated experts per offloaded layer (from a hot-list produced by a small offline profiler that counts
ffn_moe_topkselections), and at decode only it runs a dual FFN chain: hot experts from VRAM, cold from RAM, merged once per layer. Prefill routing and any unsupported shape fall through to the stock batched path (verified: identical graph splits), and it is zero-overhead when off — the prefill cost of holding the copies is real though, quantified below.Numbers on Qwen3-Coder-Next MXFP4 (80B-A3B, 10/512 experts),
--n-cpu-moe 24, temp 0, serverpredicted_per_second:--n-cpu-moe 24: 72.1 tok/sThe win is really in adaptive, per-session hot-lists: a generic corpus only covers ~33% of a live task's expert hits (→ +7%), while a short history of the user's own recent sessions transfers at ~65–69% (→ +26%). Gain scales ~linearly with covered bytes and a simple bandwidth model predicts it within ~3–5%.
I also tried to close the remaining fixed per-layer dispatch floor with async overlap of the cold chain — negative result: at
ggml_backend_schedgranularity it can't work (splits run strictly sequentially, the CPU backend has no async/event path), and forcing a split boundary to carve a join costs ~+0.98 ms/token. Details in the writeup.One honest trade-off, measured: prefill routing is untouched (identical graph splits; a parsed-but-empty hot-list is free at +0.006%), but the ~4.4 GiB of resident hot copies cost −9.9% pp2048 at HOT_N=96 — so end-to-end the split only wins when output tokens dominate the prompt (break-even ≈ prompt:output 2.3 on my config); prompt-heavy workloads should lower HOT_N or leave it off.
This is decode-only, gated-SwiGLU only, one-model-per-process, CUDA-tested only — a prototype to measure the idea, not merge-ready. I'd appreciate feedback on whether and how this is worth upstreaming (env vars → CLI flags, CPU-path validation + tests, and the general form — overlap any CPU-resident expert chain, not just a hot/cold split).
Branch: https://github.com/JigSawPT/llama.cpp/tree/aipc-v2-pr — tools, evidence and full writeup: https://github.com/JigSawPT/moe-autopilot
koren1712 commentedon Jul 4, 2026
Small data point from a Windows/CUDA fork that may be relevant to this cache design:
https://github.com/koren1712/llama.cpp/tree/expert-streaming-win
On GLM-4.5-Air Q4_K_M, a 16 GiB zero-copy slab cache reduced streamed disk bytes
from 132 GB -> 76 GB per 32-token run, but end-to-end speed dropped from 0.66 t/s
to 0.60 t/s on a 32GB RAM machine. A 24 GiB RAM tier collapsed to 0.12 t/s.
So the pinned/slot-buffer direction here looks important, but the guardrail on
resident RAM size also matters on 32GB systems.
JigSawPT commentedon Jul 4, 2026
Thanks — that's a really useful data point, and it rhymes with what I measured on the VRAM side: residency is never free, at any tier.
My setup avoids your regime entirely: I only run models whose full expert set fits RAM+VRAM (
--no-mmap, no disk tier), and the +26% only holds under that condition. Where residency did bite me was on VRAM — the ~4.4 GiB of resident hot copies cost −9.9% prefill throughput even though prefill never computes with them.So I'd generalize your conclusion: every tier needs an explicit residency budget. My working rule on Windows: keep working VRAM + system baseline ≤ ~30 GB, or WDDM starts paging silently and inverts your results — same failure shape as your 24 GiB collapse.
Curious: does the slab become a net win once the model fits RAM+VRAM, or does page-cache competition still dominate at 32 GB RAM? That boundary is exactly where the static-vs-dynamic tradeoff seems to live.
JigSawPT commentedon Jul 5, 2026
Follow-up: the split now covers the
SWIGLU_OAIactivation (per-expert bias), so it applies to gpt-oss-120B too — +31.6% decode (36.2 → 47.6 tok/s) on the same RTX 5090, VRAM-bound at ~30 GB.One honest note worth flagging for this cache design: an adaptive per-session hot-list doubled held-out coverage (28% → 45%) but bought only +2.3% tok/s here — gpt-oss routing is flat, so the extra covered experts are low-traffic tail. On a concentrated-routing model (Coder-Next) the same adaptive lever is where the win lives; on a flat one it barely moves. So per-token coverage is necessary but not sufficient — the eviction/hot-set policy this issue contemplates will pay off very differently across models.
Branch + numbers: https://github.com/JigSawPT/llama.cpp/tree/aipc-swiglu-oai
koren1712 commentedon Jul 6, 2026
Ran the fitted-regime check.
Short answer: yes — the slab becomes a net win once the touched expert working set fits. But your “residency is never free” point still shows up clearly.
I tested Qwen3-Coder-30B-A3B Q4_K_M on the same 32GB RAM + RTX 4080 16GB machine, 128-token repeat, one run each:
--n-cpu-moe 99mmapFull note: https://github.com/koren1712/llama.cpp/blob/expert-streaming-win/docs/slab_fitted_regime.md
So in this fitted working-set regime, the 8 GiB slab beats the stock mmap CPU-MoE path and cuts streamed bytes 141.5 GB -> 8.18 GB. My earlier GLM-4.5-Air slab result was outside this regime because the expert payload does not fit my 32GB RAM + 16GB VRAM machine.
The 8 GiB -> 16 GiB comparison is the useful warning: same streamed bytes and same hit count, but the larger residency is slower. Past working-set coverage, extra residency buys no fewer reads and just costs memory pressure / OS competition.
Your gpt-oss hot-list result also matches the pattern I’m seeing: raw coverage is not enough. The useful metric is weighted traffic coverage, and the optimal cache policy probably depends heavily on whether routing is concentrated or flat.
Caveat: single model, one run each. I would not generalize the exact crossover size, only the mechanism.
JigSawPT commentedon Jul 6, 2026
Nice result — and I think your setup and mine are the same cache policy at two different levels of the memory hierarchy, which is why we keep landing on the same law.
Your 8→16 GiB penalty (same streamed bytes, same hits, but slower) is the same knee as my gpt-oss ceiling: once the fast tier covers the weighted-traffic working set, extra residency buys zero fewer reads and only adds pressure. You read it as OS/memory competition; I read it as the VRAM/WDDM cliff — working set + baseline past ~30 GB on a 32 GB card silently pages and inverts the result. Different tier, identical shape.
The only real difference is which two tiers. Your slab caches storage→RAM because the expert payload doesn't fit your 32+16 GB box (hence 141.5 GB streamed with no slab). My static hot/cold split assumes the experts are RAM-resident — PCIe idles at ~4–6 MB/s in decode, so it's pure RAM-bandwidth-bound — and copies only the hot ones up to VRAM. Same admission problem, different bandwidth ratio and fit threshold, which is exactly why the mechanism ports but the crossover size won't.
On coverage we're seeing the same thing, and I'd argue the metric to standardize on is weighted traffic coverage (fraction of decode-time access mass served by the fast tier, not raw bytes or expert count). Its knee is set by routing concentration: Coder-Next puts ~80% of hits in ~28% of experts → coverage translates almost linearly (+26%); gpt-oss is flat → doubling held-out coverage 28%→45% bought only +2.3%. A cheap Gini/entropy of the expert histogram would predict where your slab stops helping before you sweep sizes. I already emit a per-expert hot-list from the profiler, so it could also pre-warm the slab — admission by prediction instead of by demand, killing the cold-start re-read spike. Single-run both sides, so it's the mechanism I'd bank on, not the exact crossover.
experiments : add slot-cache design doc + session report
memoriaru commentedon Sep 2, 2026
An update for everyone tracking this: I built and validated a working implementation of this feature — RFC is up here: #28248
TL;DR: --moe-expert-cache N keeps a persistent pool of N expert slots in VRAM per offloaded MoE weight tensor. No new ggml ops — expert ids are remapped through a map table via get_rows, so it works on CUDA/Vulkan/Metal unchanged. Two design points that came out of this thread's discussion:
Prefill never touches the pool: any ubatch that can select more distinct experts than the pool has slots falls back to the existing selective-copy path, so prefill behaves identically to master and never evicts decode state (the problem SLRU probation was proposed to solve).
Slot sizing follows the working-set rule discussed here (JigSawPT/koren1712's findings): N = top-k is a full-miss worst case and is measurably slower than master; gains start at ~2× top-k.
Numbers (RTX 4090 24GB, Qwen3.8-Flash-Next, all 48 MoE layers on CPU, Q3_K_XL): decode 11.55 → 21.21 t/s (+84%) at N=64, monotonic; prefill unchanged; perplexity-equivalent to the host-copy path (3.3096 vs 3.3262 ± 0.053); long-session anchor outputs byte-identical. Validation found and fixed a real scheduling bug along the way (evidence chain: https://github.com/memoriaru/llama-cpp-expert-pool-stale-table-fix).
PR will follow after maintainer feedback on the RFC. Thanks to everyone whose measurements in this thread shaped the design.
Btw I only want to find way to increase local graphic moe prefill speed this model at start.
XBold commentedon Sep 5, 2026
This RFC (Issue #20757) laid the groundwork for everything that's followed — the two-tier GPU+RAM expert cache concept directly inspired PR #27861's LRU implementation and influenced the broader MoE caching discussion. Thank you for articulating the problem so clearly.
For anyone following this thread now, the landscape has evolved significantly:
What's been implemented since this RFC:
--moe-expert-cache Nslots, async throttled uploads, +31% throughput on Qwen3.8-Flash-Next)--moe-stream-*flags) for models exceeding RAM, with device-side slot caching and wave-partitional prefillThe convergence question:
These implementations solve the same underlying problem — MoE experts should live where access patterns dictate, not where the GGUF file puts them — but they're still separate flags and separate code paths. The natural next step seems to be a unified residency abstraction:
This would let the engine manage expert promotion across tiers with a single eviction policy, rather than requiring users to tune
--moe-expert-cache,--moe-stream-cache, and--cache-ramindependently.For the maintainers: With 5+ independent implementations of expert caching/streaming now in flight, there's a real risk of fragmented APIs. A shared residency abstraction would benefit everyone — users get a simpler interface, and the codebase gets one well-tested implementation instead of several overlapping ones.
Thank you to everyone who's contributed to this space. The fact that a single RFC from months ago has sparked this level of innovation is a testament to the quality of this project. 🙏
JigSawPT commentedon Sep 9, 2026
Thanks for the map — it's the clearest summary of where this went. Agreed that one residency abstraction beats five flags. From the measurements behind this thread, four things the tier string
GPU:LRU:128,RAM:LFU:256,SSD:streamcannot express yet, and which decide whether a configuration helps or inverts:nvidia-sminor the shared-memory counter shows it.GPU:LRU:128is a size; what has to be enforced is "≥ 1 GB free at max context", which makes the GPU tier size an output.p_e × bytes_saved − upload_cost > 0, with the loop closed on measured inter-token time.Nper tier misallocates slots on the flat layers.Details and the static-vs-LRU numbers on the same model are in my comment on the RFC (#28248). Happy to run the unified design on this box once there is a branch.
docs : correct related-work claims after reading ggml-org#29949 in full
SurgeonTalus commentedon Oct 21, 2026
Data point from the Drama 3 port, measured on an Apple M1 Pro (16 GB, 10 cores) with a Drama 3 GGUF (40 layers x 256 routed experts, Q3_K experts, 8-bit dense backbone).
The expert read path in
DiskExpertWeightStore::stage()copies the misses of one call sequentially, so the drive is driven at queue depth 1. On this machine that yields 327 MB/s (57.3 GB read in 175.4 s), against 2-3 GB/s for the same drive on sequential reads.Issuing the misses of a group in parallel changes that. The staging ranges are disjoint, so threads cannot collide. Same model, same text, same seed,
expert_cache_size=32:RTF 25.5 -> 16.8 (-34 %), output bit-identical.
Raising the cache size interacts with this, and the interaction is worth noting for the eviction-policy side of this issue:
At
expert_cache_size=64the read and the upload are both cheap, and the wall time is 146.7 s against 102.0 s of staging. Atexpert_cache_size=96the read gets cheaper still (74.1 s) but the upload to the GPU becomes the limit, and with 16 threads it degenerates badly.So on this hardware the useful range is a cache around 64 with 8-16 read threads, and the upload path, not the read path, is what limits a large cache.
Two things that might be useful for the two-tier design in this issue:
pin_expertspath (pinned host buffer) copies the whole expert bank into host RAM at load and uploads directly from it. That is a different trade from the mmap-backed cold path, and it would change the numbers above substantially on a 16 GB machine, where the whole bank does not fit.refresh: background work, ideally hidden behind computein this issue) is where the remaining headroom is. With 44.7 s of compute in the same run as 102.0 s of staging, there is real compute to hide I/O behind, but only if the reads for the next group are issued before the current group finishes.The prototype in this repository measures a 2.09x throughput gain from double-buffering alone, which is consistent with the spread above.
memoriaru commentedon Oct 22, 2026
@SurgeonTalus this is a very clean dataset — thank you, especially for the cache-size × threads interaction table. Three readings from our side (we implemented the VRAM-pool variant tracked in #28248 — persistent per-tensor slot pools in VRAM, RFC at discussions/28248, tip
555d1ec):The upload-path inversion at cache=96/threads=16 (234.7 s) is the most useful failure in the set. On M1 Pro the "upload" is a host-RAM copy into unified memory, so 16 concurrent staging threads eventually saturate the same LPDDR fabric the expert compute reads from — adding threads past the fabric's concurrent-write tolerance turns bandwidth into contention. That's a per-platform cliff (on discrete-GPU boxes the same bytes ride PCIe instead), and it means any two-tier design needs an upload concurrency cap rather than "more threads is better".
Your double-buffering result (2.09× from staging/compute overlap alone) independently validates the async-refresh direction this issue has been pointing at. Our current mainline branch deliberately uses synchronous copies for the per-miss expert uploads — on Vulkan, async writes can land on a dedicated transfer queue unordered against the compute queue reading them, a silent hazard that CUDA never exposes. The correct end-state is what your prototype does: double-buffered residency so the copy for group N+1 overlaps compute for group N. Your 2.09× is the first measured number for that gap on the SSD tier; on our PCIe tier the equivalent headroom is the decode-speed ramp we measured as hit rate climbs.
The parallel-miss-read result (327 MB/s → ~1 GB/s effective at 16 threads) generalizes the QD1 problem to every SSD-tier implementation — including the NVMe-backed expert stores discussed elsewhere in this thread. Staging ranges being disjoint is what makes the threading safe; worth stating in the two-tier design so implementers don't ship the sequential version.
One question: is
DiskExpertWeightStorepart of your Drama 3 port's tree, or a branch others can pull? The interaction table (cache 64 beating cache 96 under contention) is exactly the kind of measurement the pluggable-eviction-policy design in this issue needs, and it would be good to link the implementation next to the numbers.For anyone cross-reading: the VRAM-pool implementation and measurements from the RFC side live in discussions/28248 (96-case correctness matrix, per-tensor telemetry columns, and a recent decode dataset on Qwen3.8-Flash-Next 512-expert where caching turned a 6.5 t/s stock path into 10+ t/s). The two efforts are the same policy at adjacent tiers — disk→RAM (yours) and RAM→VRAM (ours) — and your upload-cliff data is a direct input to the async-upload design for the upper tier.
Helldez commentedon Oct 26, 2026
My project is precisely of this type and is based on llama.cpp.
Working toward a unified streaming framework - from phones to NVIDIA GPUs - that supports any model, any hardware, and any OS. Automated resource scheduling, tailor-made to optimize tok/s.
Perhaps this might help:
https://github.com/Helldez/BigMoeOnEdge