Repository navigation
Offloaded-MoE prefill leaves the GPU idle waiting on serial expert H2D copies (single-GPU -ncmoe) #25859
Description
Sharing a performance finding from profiling the offloaded-MoE path (--n-cpu-moe / -ncmoe) — the setup where a big MoE model runs on a small GPU with the experts kept in system RAM and streamed in per layer.
My box: RTX 3060 12GB, Ryzen 5600X, 32GB DDR4, PCIe 4.0, Ubuntu 24.04. Model: Qwen3.6-35B-A3B (Q4_K_M), -ncmoe 26, prefill at pp2048 / ub2048. Everything below is single-box and gen4-specific, so numbers will move on other hardware.
On this setup prefill is bottlenecked by PCIe transfers, not GPU compute. Profiling with Nsight, the GPU sits idle a large fraction of each prefill pass waiting on expert-weight H2D copies, and those copies run serially with the matmul that consumes them — layer N's weights finish uploading, then layer N computes, then N+1's weights start uploading. The upload and the compute don't overlap on the offload path.
Two changes helped a lot, both prefill-only and both producing token-identical output at temp 0:
- Page-locking the mmap-backed CPU expert weights so the H2D transfers DMA directly instead of going through a pageable bounce buffer. ~+21% on its own.
- Overlapping the expert uploads with compute on a second stream with a few staging slots, so layer N+1's weights upload while layer N is still computing. This is where most of the win is.
Together that took prefill from ~1143 to ~1880 tok/s at identical settings — roughly +64%, pure code, no config change. GPU idle during prefill dropped from ~42% to a few percent.
I prototyped both behind env flags on a fork so I could A/B them on the same binary and confirm the output stayed identical: https://github.com/thecodacus/llama.cpp/tree/fable5/prefetch-experts (there's a benchmark writeup in the branch with the exact commands and the full before/after log). I'm raising this as a finding rather than a PR — partly because AGENTS.md asks for that, and partly because the right way to expose something like this upstream is probably whoever owns the CUDA scheduler deciding on the shape, not me dropping a patch on you.
Two honest caveats: this was measured a few weeks back at around the b6xxx era, and the CUDA backend has moved since, so the exact idle percentage may be stale — I haven't re-run against current master. And I know pipeline copies are already a thing for the multi-device case; the gap I'm describing is specifically the single-GPU offload path, which always gets n_copies = 1.
Main question for maintainers: is reducing GPU idle on the offload path something you'd want addressed, and if so, is overlap-on-a-second-stream the right approach — or is there a reason the offload copies are serial today that I'm not seeing? Happy to share nsys timelines or run other configs if that's useful.
here is nsys trace
Activity
thecodacus commentedon Jul 18, 2026
One more detail that belongs here, because it changes how you'd approach a fix.
The offload path's selective expert copy needs the routing ids on the host to know which experts to upload, so each layer does a D2H readback of the ids before the copies start. That readback is a hard sync point: compute has to finish producing the ids, then the pipeline stalls while they come back, then uploads begin. Even with pinned memory and a second stream, that sync alone keeps upload and compute from overlapping cleanly.
The thing is, at prefill batch sizes the selection isn't buying anything. I instrumented the router: at pp2048 the union of experts selected across the batch is essentially all of them (256/256 for Qwen3.6-35B-A3B on nearly every layer). You pay the sync to learn "upload everything."
So the overlap change in my branch does the obvious thing: past a batch-size threshold it skips the ids readback entirely and streams each layer's full expert tensors. That's what lets the second-stream prefetch run ahead of compute with no sync in the loop. At decode-size batches the selection still wins (only a handful of experts per token) and that path is untouched.
Worth knowing if anyone picks this up upstream: the fix isn't "make the readback async," it's "notice when selection is pointless and don't do it."
am17an commentedon Jul 18, 2026
This is the same as #21067
thecodacus commentedon Jul 18, 2026
awesome.. I didn't see the PR. hope it get merged soon
thecodacus commentedon Jul 24, 2026
opening it for now, will close it once the PR merged
yalun753 commentedon Aug 8, 2026
mobilinkd commentedon Aug 14, 2026
@am17an @thecodacus These overlap but target different cases; they have very different performance profiles when prefetching expert tensors. With DS4F_IQ2_XXS, #21067 reserves about 1.7GB of extra memory, and with a single copy slot (n_copies=1) it rarely keeps the pipeline from stalling. #25859 lets you tune the ring buffer as a multiple of the largest expert tensor. In practice 21067 is worse for me with pw=1: it pays to fetch the full weight tensors for every offloaded layer (dense and MoE alike), but the pipeline is mostly stalled. The tunable ring in this PR is the part that matters. With GGML_SCHED_PREFETCH_EXPERTS=4 using this code I can keep the pipeline from stalling and get a 17-20% prefill throughput improvement, at a cost of 2.4GB. I am curious whether this was made adjustable and defaulted to 3 slots because the same issue was encountered during its development. The problem with doing this in 21067 is that its reservation unit is a whole fused split (all of a layer's offloaded weights) rather than a single expert tensor, so adding ring slots there is several times more expensive per slot than the expert-tensor-sized ring here.
There is real value in distinguishing prefetching expert weights from prefetching any model graph.
Note: I had to adapt the prefetch to work around SYCL, and what has worked is an expert-only prefetch modeled on this code -- in part because the ring size is tunable. This comes out of SYCL work I've been contributing via @arthw.
yalun753 commentedon Aug 14, 2026
This matches what we measured independently on our expert-offload path (experts mmap'd in host RAM, GPU reads them via PCIe DMA). Point 1 (page-locking) is confirmed as the real win: we already pin expert tensors with cudaHostRegister — selectively, only the top-K experts per layer from a router map, with on-demand pinning as fallback for the rest — and measured ~0.43 ms page-lock on first touch.
On point 2 (overlapping H2D upload with compute on a second stream): we tried exactly that (double-buffered multi-stream pipeline) and it measured no benefit, or a regression, on our setups. Single expert tensors are small (e.g. 1.55 MB ≈ 60 µs over PCIe Gen4 vs <10 µs kernel time), so there's nothing for the overlap to hide; on DeepSeek-V4-Flash (43 layers × 256 experts, very scattered routing) staged copy was actually slower. Instead of overlapping the copy we remove it: hot experts stay resident in a VRAM LRU cache, cold ones stream in once through the DMA path. That also sidesteps the TTFT regression and the permanent double-buffer VRAM reservation that whole-tensor prefetch (#21067) pays on MoE.
Measured end-to-end (moe-l2, llama.cpp fork): Qwen3.6-35B-A3B 30.9-48 t/s, DS-V2-Lite 85-133 t/s, DeepSeek-V4-Flash 157B 10.1 t/s @ 17.4 GB VRAM (85 GB file, 2080 Ti/4090). Details in our issue #26448.
Castoff995 commentedon Aug 27, 2026
Additional Windows / AMD RDNA4 / Vulkan datapoint for the stock single-GPU
-ncmoepath.I spent some time tuning Qwen3.6-35B-A3B on an RX 9070 XT 16 GB and found a fairly sharp interaction between
-ncmoe, batch/ubatch size, and KV quantization. No code changes or custom fork were used.Environment
b106440.3.0-devd7a207411Qwen3.6-35B-A3B-UD-Q4_K_M.gguf(Unsloth, ~22.1 GB)--parallel 1llama-servertiming output, notllama-benchFinal configuration
48.5k real-context result
The synthetic prompt contained 48,531 input tokens and generated 256 tokens.
ncmoe=22,b=1024,ub=512, Q8/Q8 KVncmoe=20,b=8192,ub=2048, Q4/Q4 KVSo configuration tuning alone changed:
-ncmoehas a sharp performance boundaryWith the final
b=8192,ub=2048and Q4/Q4 KV:-ncmoeThe
20 -> 19step alone drops generation by about 16%.I am not claiming a cause for this boundary; I did not profile PCIe transfers or GPU idle time. I am only reporting the reproducible behavior.
A repeated fresh-server run at
ncmoe=20reproduced the same prompt-processing progression and exactly 42.01 t/s at the 128-generated-token checkpoint, so the jump at 20 was not a one-off.Batch / ubatch sweep
At 48.5k input tokens, before switching to Q4 KV,
ncmoe=22showed a strong batch-size effect:b / ubOn this particular 16 GB RDNA4 setup, increasing logical batch helped substantially up to around
b=8192, while making the physical ubatch too large was actively harmful.ub=3072was also tested. After the finalncmoe=20 + Q4 KVconfiguration it regressed to 1466.00 PP / 38.98 TG versus 1565.65 / 42.18 atub=2048.KV-cache interaction
At
ncmoe=20,b=8192,ub=2048, with 48.5k input tokens:The interaction between KV type and the useful
-ncmoeplacement was large enough that tuning-ncmoein isolation would have missed the best configuration on this system.Again, I don't want to speculate about the exact mechanism without profiling.
Near-full 128k context
I also tested a 121,269-token prompt, close to the configured 131,072-token context.
With Q4/Q4 KV:
With the identical prompt using Q8/Q8 KV:
Generation lengths differ because the model's reasoning ended at different points, so I would not compare total request time directly.
Q4 vs Q8 long-context retrieval check
Because Q4/Q4 was faster, I also checked whether the lower-precision KV cache caused an obvious retrieval failure.
The 121,269-token synthetic test contained:
PROJECT -> TAG -> secret, with the two records placed far apart;Results:
Both configurations returned every requested value exactly and rejected all of the decoys.
This obviously does not prove that Q4 KV is quality-equivalent to Q8 KV in general. It only shows that I could not measure a recall/association loss in this particular ~121k synthetic retrieval test.
--load-mode noneEarlier in the tuning process, using CPU tensor overrides with mmap produced:
Following that recommendation mattered on this machine.
In an early 32k-context test at
ncmoe=20, the same short generation improved from about 27.6 t/s with mmap to about 31.4 t/s with--load-mode none, with a much larger improvement in prompt processing.Why I'm posting this here
This is not a diagnosis of the serial-H2D behavior discussed in this issue; I did not capture a Vulkan transfer timeline.
I thought the numbers might nevertheless be useful as a current stock-llama.cpp baseline for the same Qwen3.6-35B-A3B Q4_K_M offloaded-MoE workload on:
-ncmoeThe large sensitivity to
b/ub, KV type, and especially the sharpncmoe=20 -> 19cliff may also be useful when evaluating future pinned-memory / transfer-overlap changes: a default-ish configuration can leave a substantial amount of performance on the table before any code-level optimization is measured.Happy to rerun exact configurations or test a patch on the RX 9070 XT if useful.
AI assistance was used to organize this benchmark write-up; all performance numbers above are copied from tests I ran locally and checked against the console logs.
Retune the Qwen3.8-Flash-Next dual-GPU entry for offload-bound prefill
Sophia-Thickums commentedon Oct 5, 2026
Independent datapoint on AMD / Vulkan (Vega-class, gfx900) — under-represented in this thread. It confirms the router observation in @thecodacus's second comment, with numbers behind it.
Setup: 2x 16GB AMD cards (Instinct MI25 + Vega Frontier Edition, both gfx900), 30GB DDR4, llama.cpp Vulkan backend. Model Qwen3-30B-A3B Q4_K_M (~17.3 GB). Single card, -ncmoe 12 baseline.
Confirmed by interleaving ub512/ub4096 ABAB at -r 5, cold each time — not a single warm sweep: [75.8, 77.9] vs [156.5, 156.3]. Decode unchanged (~50 t/s). Above -ub 4096 it plateaus (6144/8192 -> ~151).
This lines up exactly with @thecodacus's router instrumentation: at large prefill batch sizes the union of selected experts is essentially all experts, so the per-layer selective copy buys nothing — you are paying H2D overhead plus the D2H ids readback sync once per small chunk, for a selection that would have been everything anyway. A bigger chunk amortizes both.
-ncmoe is not monotonic — there is a fit cliff. At -ub 4096, sweeping -ncmoe: 12 -> ~50 t/s decode; 8 -> 10.5 t/s. Less offload was catastrophically slower: at ncmoe 8 too many experts stay on the 16GB card, it stops fitting, and decode collapses. Not a speed curve — a fit boundary. Worth sweeping rather than assuming monotonic. (Matches the general "sharp interaction between -ncmoe and ubatch" that @Castoff995 reported on Windows/RDNA4.)
The offload can be eliminated entirely when the model fits across cards. With the whole model resident on the two cards (-ncmoe 0, split ~9.6/9.2 GB), no expert is non-resident, so nothing crosses PCIe:
Caveat that bit us: the pair only helps with -ncmoe 0. With any offload, the two cards share one bottleneck and read as a single card. This is the endpoint of "@thecodacus: the selection isn't buying anything" — if the model fits whole, there is no per-chunk transfer to optimize at all.
Full writeup + raw logs (including the runs that failed, which is where the device-selection and -c traps showed up): https://github.com/Sophia-Thickums/vega-moe-pool
One tooling note for anyone reproducing #3: llama-bench cannot reproduce the -ncmoe 0 config — it has no -c flag, so it runs at a huge default context, overflows the cards, and returns radv/amdgpu: The CS has been cancelled because the context is lost (a GPUVM fault that looks like hardware). Use llama-server, which accepts -c.