Repository navigation
llama : add a GPU cache for MoE experts kept in host memory - #29887
Conversation
github-actions
Bot
added
the
ggml
|
0.00.345.120 I common_params_fit_impl: getting device memory data for initial parameters: From the original https://github.com/tetherto/qvac-fabric-llm.cpp
@am17an Why did you chose to invest your time and work into something that declares to not support OpenCL, tensor parallelism, multi-GPU execution, and training? There are dozens of expert cache PRs and forks on Github that do run on multi-GPU systems and have been waiting for many months for a code review. |
|
Because I want something that can be reviewed and merged? |
|
Sounds promising. Is there any plans to add multi-gpu support in the future? |
|
Tested on AMD (ROCm/HIP, Windows) at Box: Radeon AI PRO R9700 32 GB (gfx1201), Ryzen 7 8700F (8 cores), 32 GB RAM, Windows 11.
The cache builds and runs on HIP ( Two things from reading the diff:
Am happy to run tests again in the future as this PR moves :) AI disclosure: I used an AI assistant to run the benchmarks and draft this comment. I checked the numbers against the logs. |
|
Would it be good to add an argument to control the number of layers that participate in the cache, like For a very limited VRAM budget (e.g. ~5% of total expert size, boundary depends on hardware), hit rate could be low. The process of uploading a layer's routed experts through PCIe + computing them on GPU could be slower than just computing all the experts of that layer on CPU. In this case, specifying only a portion of host experts to participate in cache could increase the hit rate for higher performance gain. |
|
I've been turning the idea of an moe cache over in my head for a while too. Haven't had a chance to look at this implementation in detail yet, will try later today. One idea is to reuse the compute buffer for caching during generation. It's sized for prompt processing which is way more than is needed during generation so is potentially free cache space for tg. The cache (or the portion in the reused buffer) would need to be wiped each time we swap between pp & tg but it should warm up pretty quickly once generation starts again. I suppose you could store the cached expert indices when pp starts then reload them when swapping to tg. The other idea was to update the cache asynchronously in a lagged fashion. On a cache miss we still process that expert on the CPU but queue up a transfer into the GPU cache. If hit rates are sufficiently high it should still be beneficial for the next token (on average) and let's overlap the transfers with compute for the current token rather than holding up computation. Need to take a closer look to see how you're handling it here.
I think that's a good idea, how beneficial the cache is will depend heavily on hit/miss rate so limiting the cache to specific layers is one way to push up hit rates. A raw number of layers is a good simple knob to adjust, might be worth adding a way to select which layers are cached (a regex like |
I've tried pretty much that in my fork, combined with the frequency-based admission and eviction (simple LRU performs very poorly). This got me around 7-8Gb (with MTP) for caching on the 20Gb of VRAM budget (4070+3070) with 2048 to 3072 buffer size. On the 4K_M quant of qwen3.8-flash-next it translated to around 60% cache hit ratio and 200-400 t/s prefill and 35-20 t/s decode depending on the context size. |
|
This implementation being backend-agnostic is a plus. Seems like a good starting point and can be further optimized in later commits/PRs. |
I have been getting bombarded with ideas and recommendations. This seems like a feature that everyone wants and I can't claim that my implementations do things better as I've seen at least 1 that does at least 1 part better, but can be behind in others, ideally we need to organize all PRs into 1 taking the best (compatible/adapted) ideas from each. I already have a local basic document I strictly manually maintained comparing the ones I tested so far vs mine in code and final result and I could make a version that is the best of all world's, but I'd rather directly collaborate with people. I don't know what the right place is to organize that at as I only have their github usernames |
|
Just to be clear: I haven't seeked out any help regarding this and probably won't be looking at any other PRs for this feature. |
Damn you are much better than me at sticking to scope. I started with the same logic but then ended up making it unmergable, my first iteration and your current one are almost identical feature wise but with different priorities |
|
@am17an I'm running an unusual configuration - multi-GPU, but the second GPU runs only experts (-ts 1,0 + -ot "blk.(32-47).ffn_*_exps=Vulkan2"), everything else, including the KV / indexer / RS buffer, is on the first GPU. On top of that, -ncmoe 30 keeps the experts of layers 0–29 host-resident, so the layers this cache would actually manage all have their MUL_MAT_ID nodes computing on the first GPU, which is where the cache would attach. At first glance I don't see why --moe-cache-mib shouldn't work here (not saying I'm 100% confident and happy to be told I'm missing something). |
|
@am17an it does work with my dual GPU weirdo setup! Got up to +40% decode speed - great job. May I ask not to rule out gpu > 1 entirely with a throw but maybe leaving an option (with ENV?) to bypass it to a warning? This will make life so much easier when it lands in main. |
|
@sasa7812 glad it works for you! I just want to keep it as simple as possible for now. Everything will be added eventually with a good test coverage. |
|
@am17an If possible, can you add a benchmark for a smaller MoE model (say Qwen 35A3B) that fits entirely on GPU, so we can compare the difference between full vs partial vs no offloading? |
I may be misunderstanding your idea, but it should if i understand the question right not give any extra speed to full fit. Tho not heavily looked into the details but dont see why it would and know my implementation wouldnt |
Far off as in the fraction, not the multiple. No offload (GPU only) is 1.0 i.e. the theoretical max. With offloading or caching, we enter fractional territory. Then in later PRs, this fraction can be increased to get closer to max throughput (1.0). |
OH its a bench marking number question. you want to see fully on gpu vs partially offloaded even if it fits. Got it. I am doing a big consolidation of all implementations, reading all the code and testing them to find each ones strengths. If someone else hasnt ill share my results to this test when i get to it for both his and mine (and others if you want) obviously if someone responds prior to me i wont need to |
Qwen3.8-Flash-Next is a great test for these kinds of things |
I tried comparing lfm 8b with pure fit vs 50% cached. it reached 90% of full fit of stock. however looking into it deeper, LFM repeatedly reactivated the same couple experts so its expected, I do not have the hardware to test (locally, could use cloud but wont for now) on a real model people use as my setup cant fully fit it for a baseline |
…d direct lazy-row reads (upstream ggml-org#29030) - llama : add a GPU cache for MoE experts kept in host memory (ggml-org#29887, port of the qvac-fabric cache). --moe-cache-mib / llama_context_params::moe_cache_size: MUL_MAT_ID on host experts runs on the GPU from an LRU cache, only misses are uploaded, for batches of up to 32 tokens. Single device only. - qwen4exp/gemma4: gather lazy tensor rows with direct reads (ggml-org#29030, supersedes ggml-org#28136), -lzm on. Measured on a whole Qwen3.8-Flash-Next UD-IQ3_XXS (78 GB) with one RTX 5060 Ti 16 GB, 62 GB RAM, --cpu-moe: generation 14.95 -> 19.26 tok/s with --moe-cache-mib 8000, prompt unchanged (~89 tok/s). -lzm on was slower on this machine (79.5 / 16.38). The expert prefetch of ggml-org#28414 was left out: with it the prefill aborted in cuBLAS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ne stage to size it (#280) llama.cpp can keep the most recently used experts of host-resident MoE layers in a GPU cache (ggml-org/llama.cpp#29887, split over GPUs since #30112). Add it as a model setting, written only beside CPU expert layers and only for builds from b11507. The VRAM estimate counts it, capped at the experts it caches, and places it on the cards that hold those layers. The fit planner leaves it off, since it owns GPU memory, and Autotune gains an expert-cache stage: each batch finalist with the cache off and at a quarter, a half and all of the room left beside the profile. The setting is also a benchmark sweep field.
|
Does MOE caching support RPC? |
This comment was marked as low quality.
This comment was marked as low quality.
|
@Volunteer-1 the results in the PR are with PCIe-4. I think in general you have a poor understanding of benchmarking and should not be commenting or guiding other users. I will mark your responses as "low quality" from now on. |
|
Oh so THIS is the commit that completely broke my whole project. @Volunteer-1 is right... |
|
@Atomic-Germ I'm quite curious how an optional feature "completely broke" your project? |
|
I should have been more specific. Hopefully someone has a similar setup can confirm. I'm running Qwen 3.8 Flash Next Q4 UD K XL using RPC for one of 2 use cases and to evaluate performance.
For #2, I was hoping to use this PR to cache MOE since they obviously can't fit on a 24GB GPU. The way I configured is using OT where RPC. ngrams and MOE in ram on 2nd machine with draft model and other layers in VRAM. Release: b11721 It's possible I missed something so I'll look over it in the meantime. Thanks. Additional Notes: Disabled RPC and tested on one machine. 5950x, 128 DDR4 and dual R9700. One-shot tests with short prompt, TG only. ROCM + layer split
ROCM + tensor split Vulkan + layer split Single GPU on machine 1 isn't useful though I suppose I could test the RTX 3090 to see if it benefits without RPC. |
|
@d-shehu please give information how you connect the two machines.
One user reported that this PR consumes on average 20.9 GB/s PCIe bandwidth or 24.7GB/s up to maximal 35 GB/s PCIe bandwidth with GLM-5.3-Flash UD-IQ3_XXS and breaks even with CPU-only inference at 85% cache hit rate. Comparison with theoretical maximum full duplex PCIe bandwith: |
|
Folks, there are so many things beyond maintainer's control that could go bad and make it slow. For example, you exceptionally large GPU could be sagging and break the contacts just enough to downgrade it from x16 to x8, or worse. Heck, this very person writing this had to re-seat their card, because apparently I was too gentle during the initial assembly. |
…#29887) * llama : add a GPU cache for MoE experts kept in host memory Assisted-by: Claude * use llama_moe_cache_ptr
|
Here is a Powershell script that calculates the maximum theoretical decode speed for an LRU cache for any open weights MoE model based on the model's meta data, the user's choice for bits per weight for the experts and several PCIe configurations, and compares it to CPU-only decode. Only data transfer bandwidth is used as physical limiter without time spent for calculations, therefore it's a theoretical maximum speed. The shared experts are assumed to be VRAM resident. Call it like: SPOILER: |
276 upstream commits. Conflicts and how they were resolved: - K2-Horizon: upstream added its own port of the same ifm-ai code (ggml-org#29535), with the chat parser, the unicode splitter and tests. Upstream's model, converter and names are taken; GGUF names are identical, so existing K2-Horizon GGUFs still load. The fork keeps only the K2-Horizon-36B tokenizer hash, which upstream dropped. LLAMA_VOCAB_PRE_TYPE_K2_HORIZON moves from 60 to 61 to follow upstream (it is not stored in GGUFs). - MoE: upstream moved the selective expert copy out of the scheduler into a llama_context copy callback (ggml-org#29943) and added a GPU cache for host experts (ggml-org#29887). The fork's partial mul_mat_id (id < 0 rows, used by LLAMA_MOE_HOT) now skips negative ids in that callback. build_lora_mm_id takes both the fork's partial/ids_s and upstream's MoE cache slots; build_moe_ffn_exps passes the slots on the untiered path only, so the MoE cache and LLAMA_MOE_HOT tiering do not stack on one layer. - mmvq: upstream's shared-expert fusion and the fork's negative-id early return combined; the shared-expert block never takes the early return. - server: upstream's rejection sampling and checkpoint replay paths gained the fork's on_sample hook, so speculative logprobs cover them too. - qwen4exp / build_attn: turbo Q pre-rotation kept on top of upstream's reworked QSA mask and the new training path. - top-k.cu: upstream's CCCL >= 3.4.3 gate; the fork's hipcub arm was dead since HIP stopped defining GGML_CUDA_USE_CUB. - ggml-cuda init_tensor: upstream's async memset on owned buffers, the fork's host-side padding clear on externally owned ones. - tests/test-turbo-kv.cpp, tests/test-moe-tier.cpp: moved to common_batch and llama_process, as upstream removed common_batch_add. Tested on gfx1100: ctest 76/78 (the two failures are environmental: the ggml-vocabs clone has git-lfs pointers, and system python has no jinja2; test-jinja-py passes with it). test-llama-archs 506/507; the qwen2moe "Meta" failure is a seed-dependent flake that the pre-merge build shows too. Server speculative/logprobs pytests 18/18. Needle at 12k tokens with turbo4 KV: K2-Horizon-36B, Qwen3.8-Flash-Next, Qwen3.8-27B (plus MTP).
Allow configuring the physical batch size of the draft context. The drafters feed it at most the drafted tokens per sequence and chunk larger batches by its own n_ubatch, so it does not need the ubatch of the target. A smaller value shrinks the draft compute buffer and leaves more VRAM for the target KV cache (the draft is fitted together with the target). The default 0 keeps the current behavior of following --n-ubatch. Ported from upstream commit 3fb4767 (ggml-org#30459): 5 conflicts resolved by keeping both the fork's --spec-draft-n-min-adaptive and the new option; the MoE cache line in speculative.cpp belongs to unapplied ggml-org#29887 and was excluded. Staged stat matches upstream exactly (9 files, +63/-3). Verified: build of llama + test-arg-parser OK, test-arg-parser all tests OK.
Generally bad practice to comment on already done PRs, but if it was found that a percentage is the way to go, it might be worth considering allowing to pass a percentage for that flag, maybe? |
|
We can add helper flags |
Why? It would make it very non user friendly. Let's say you have (and it is easy to see) 10 GiB of spare VRAM, you know how much to give for the cache no problem and the rule is simple - the more, the better, no exceptions almost. Now, imagine you need to find out the total size of your experts and only those that are CPU resident. And people probably trying different quants of the model. |
…#29887) * llama : add a GPU cache for MoE experts kept in host memory Assisted-by: Claude * use llama_moe_cache_ptr
Port of the qvac-fabric MoE cache. MUL_MAT_ID ops on host experts run on the GPU with an LRU cache of experts, only the misses are uploaded. Small batches only (<= 32 tokens), large batches use the regular offload path. Layers with different expert layouts get separate banks.
Enable with --moe-cache-mib N. In experiments it was found 10% of the total expert size is good.
Performance
Qwen3.8-Flash-Next Q4_0 (93.7 GiB, 65.4 GiB of experts), SPEED-Bench
qualitative(11 categories x 3 samples,--osl 512, temp 0), no MTP.llama-server -fa on -c 32768 -b 2048 -ub 2048 -t 16, CPU on the GPU's NUMA node (EPYC 7742, 16 cores), PCIe 4.0 x16.Baseline is master with the default
--fit. With--moe-cache-mib,--fitcounts the cache and keeps fewer whole expert layers on the GPU.--fit--fit --moe-cache-mib 6544-cmoe --moe-cache-mib 11000--fit--fit --moe-cache-mib 6544-cmoe --moe-cache-mib 19000Overview
Additional information
Requirements