[John-194]

Overview

For dense models, --fit moves whole layers to the CPU -ngl style, which is slow. This PR adds prioritization of what gets put on the CPU.

The plateau is because the output tensor is large, so it leaves a lot of slack for context, at the same tensor placement.
The drop at the end is where the regular -ngl kicks in and performance nears the original behaviour.

Additional information

This PR is a follow-up to my issue #27987, where doing the ordering automatically in --fit was suggested. Through testing I learned about other possible optimizations, so this PR includes a bit more than the issue proposed (like "output tensor" or "rest of layer" on the CPU, not just FFNs).

--fit ordering

Largest FFNs go first, then the rest of those layers (attention weights, norms, recurrent state) in the same order, then the output tensor, then the MTP layer, and then -ngl gets lowered only as a last resort, which causes the usual performance drop.

Ordering cost

Measured on Qwen3.8 27B UD3-Q4_K_M at 60k context as tg cost per MiB moved to the CPU (-ot, one at a time):

lever	µs/token per MiB
FFN tensors, K-quants	12
FFN tensors, IQ3/Q3 (the smallest ones)	16.5
rest of a layer, FFN already off	13.5-15
output tensor, no MTP / MTP on	12 / 13.5-23
MTP layer FFN	15-20
-ngl (scales with context length)	25-33

• Sorting FFNs by size is close enough to sorting by cost. The smallest FFNs are the most expensive per MiB, so they end up last anyway.
• FFNs before a layer's remainder: 4-9% faster than moving layer by layer.
• The output tensor is large (1 GB on both model quants) and cheap without MTP, but the MTP draft runs it once per draft token, so it can get expensive, so I'm placing it after the layer remainders, but before MTP and -ngl. For tied embeddings (Gemma) it moves the GPU copy of token_embd.

MoE fit ordering is unchanged.

The fit takes up to about twice the test loads of master, ~1.5 s for me.

Testing

I have 1 GPU, so I added a guard against multiple GPU systems because I couldn't test that. Multiple GPU systems keep the previous -ngl behaviour. If someone tests this on multiple GPUs, the guard can be removed.

Checked 18 GGUFs (Qwen 3.x dense/MoE, TinyLlama, phi 2, gemma 2-4,
mamba, rwkv 6, OLMoE, falcon h1, LFM2, MiMo) via llama-fit-params to make sure the ordering holds.

Loaded and filled Qwen 3.8 27B and Gemma 4 12B as seen in the charts.

Not PR related issue that could affect performance

--fit incorrectly calculates free VRAM. For me, the correct --fit-target values are between -400 and +200 (depending on the model and MTP use) to fully utilize my VRAM and not OOM crash upon use. Rough workaround: run the server, give it a big file to chew on, tell it to write a story, if it OOM crashes then raise --fit-target, if it doesn't then lower the value till it does, then raise it a step back.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, I used it to write the code, but I made the decisions and reviewed it.

[JohannesGaessler]

Sorry, but if this PR does what I think it does I consider it out-of-scope. The number one priority for the --fit code is robustness. For the multi GPU logic in particular I do not want to touch the basic principle that weights are assigned as contiguous blocks.

[John-194]

Sorry, but if this PR does what I think it does I consider it out-of-scope. The number one priority for the --fit code is robustness. For the multi GPU logic in particular I do not want to touch the basic principle that weights are assigned as contiguous blocks.

The PR leaves multi GPU path untouched via the last commit.

What the code does is similar to how MoE fits (FFN tensors to CPU while KV stays on GPU), though I order the layers largest first vs MoE's back to front.

The current implementation of fit makes dense models very slow for anyone needing CPU's help to run them (as you can see in the charts). My goal is to make models like 27B usable for people having <16GB VRAM.

How could we make this more robust? The simplest form we can do is by putting layers (same as ngl) on the CPU, while at the same time leaving KV on the GPU (basically ngl but without the KV part). That should fix most of the degradation, but will leave out some performance.

[John-194]

This current PR is probably more complicated than it needs to be at 207 insertions and 72 deletions. Here are simplified and possibly more robust versions, both better than master's implementation:

• mirror moe - back to front FFNs on CPU (insertions - 52 / deletions - 48), then do ngl;
• front to back whole layers on CPU excluding KV, then ngl (insertions - 73 / deletions - 52).

Here's a performance comparison:

Any suggestions?

[JohannesGaessler]

Look, I'll be straight with you: as of right now I am completely overloaded when it comes to code review. I consider this PR to be low priority vs. the other things in my backlog and based on the first review I think reviewing this PR will be more work than trying to implement something equivalent myself.

[John-194]

Can I open up a new PR that is very simple and still improves performance by a lot? I would like to mirror the MoE's implementation but for dense models, dropping all the complexity of this PR.

[JohannesGaessler]

No, because the code itself is only half the equation. I'm not convinced that the approach you're using is generally applicable to all or most setups rather than just your own and properly determining that will be where most of the effort will be needed.

[John-194]

Well, people liked my --n-cpu-ffn addition #26622 which proves that this is applicable to other setups. My goal was to optimize it, but I was redirected to --fit via my proposal issue #27987. --fit also allows automating the process.

I would have preferred being told that this is a no go in the issue before I wasted time trying this.
Anyway, hopefully someone will eventually find my charts useful and see the potencial in optimizing dense model performance with CPU's help.