[ggerganov]

Just hot loading the experts for matrix multiplication is enough to heat-up the caches. No need to add extra GGML_OP_ADD nodes for aggregating the results.

[ggerganov]

We should probably bump this up to 8u*model.n_tensors() just to be safe.

[slaren]

If I understand correctly, the motivation of this change was to ensure that all weights are loaded into memory when using mmap on a NUMA system. This would effectively revert #11571.

[ggerganov]

I think the experts are still mapped continue to be loaded because when we run the ggml_mul_mat_id() calls, we use the large n_expert_used == hparams.n_expert, instead of the original hparams.n_expert_used. So for example this call, during warmup would still load all the experts into memory and perform the warmup:

llama.cpp/src/llama-graph.cpp

Lines 867 to 870 in 033b306

	ggml_tensor * up = build_lora_mm_id(up_exps, cur, selected_experts); // [n_ff, n_expert_used, n_tokens]
	cb(up, "ffn_moe_up", il);

The change only removes the summation nodes that sum together the obtained results for each expert. Those do not involve reading data from the model, but contribute to many number of graph nodes.

---

For reference, here is the n_expert_used initialization:

llama.cpp/src/llama-graph.cpp

Lines 512 to 514 in 033b306

	n_expert (hparams.n_expert),
	n_expert_used (cparams.warmup ? hparams.n_expert : hparams.n_expert_used),
	freq_base (cparams.rope_freq_base),

Edit: fixed wording at the start for clarity