fork-knobs.md

Fork options: flags, --moe settings, tuning knobs, environment variables, state file

Everything this fork adds to stock llama.cpp. Settings you give are used as given and are never auto-tuned. Defaults are those of
the release this file ships with.

What the fork does, in one minute

A mixture-of-experts (MoE) model has many small "experts" in every layer and uses only a few of them (for example 8 of 256) for each token. When the model is
bigger than the GPU's memory, upstream llama.cpp ("stock") keeps some layers on the GPU and computes the others on the CPU. This fork instead keeps the
experts in RAM and turns the free VRAM into a cache: in each layer a number of slots on the GPU hold the experts used lately, the GPU computes the
cached ones and the CPU the others at the same time, and an expert that keeps being used replaces one that is not (it is uploaded over PCIe). An expert that is
needed and cached is a hit; the hit rate is the share of such experts.

With no switches at all: the cache is on only when a MoE model does not fit in VRAM, the slots are sized from the free VRAM, and the fork measures token
speeds and tunes its own settings (autotune) and picks the cache layout or the stock layout (placement) for the model and the GPUs. What it learns is kept in the
state file. Everything below is for overriding that, and a setting you give is never touched by the tuner.

Which switch do I want?

I want	type
exactly upstream behaviour, nothing of the fork	--fork off
the cache, but fixed settings and no self-tuning	-at off (and --moe slots=N for a fixed slot count)
a reproducible benchmark	--moe state=0 so nothing learned earlier is used, plus -at off or explicit --moe settings; change one thing per run
the cache even where the fork would choose stock (or the other way)	--moe mode=cache (or stock)
the fork to forget what it learned	delete ~/.cache/llama.cpp/moe-state.ini, or run once with --moe mode=retest
to check whether the cache is what slows my machine down	run the same command with --fork off and compare
big pinned models to start faster	the section on loading pinned weights, below

Command-line flags

flag	what it does
--moe KEY=VAL,...	the expert cache settings and knobs, below
--moe-expert-cache N	expert slots per layer in VRAM; -1 sizes them from free VRAM, 0 = no cache (the rest of the fork stays on; use --fork off for stock behaviour)
--prefetch-experts-slots N	staging slots for host-to-GPU prefetch of big batches (same as --moe pf-slots=N)
-at on|off, --autotune	self-tuning of cache knobs, thread counts and the cache-or-stock placement; off = fixed defaults, nothing measured or saved (same as --moe autotune=0)
-lm pin|mmap|dio, --load-mode	pin: weights in pinned RAM (the default of every tool when part of the model stays in host RAM and the model fits in the RAM available now; faster prompts and uploads, the load takes longer; -lm mmap opts out); mmap: memory-mapped, for models bigger than RAM; dio: direct IO
-md FILE --spec-type draft-mtp	MTP draft head (Qwen3.8-Flash-Next, GLM-5.3-Flash), see the README
-t N, -tb N	fixed decode / prompt thread counts (otherwise tuned)

Turn the fork off and run stock

you want	pass	effect
plain stock behaviour	--fork off	no expert cache, nothing tuned, measured or saved; the same placement and kernels as upstream llama.cpp
the cache, but no self-tuning	-at off (or --moe autotune=0)	the cache works with fixed defaults, placement uses a static rule, nothing is measured or saved
stock placement chosen by the fork	--moe mode=stock	skips the cache-or-stock measurement and uses stock
the cache forced on	--moe mode=cache	skips the measurement and uses the cache
a fresh start	--moe state=0, or delete ~/.cache/llama.cpp/moe-state.ini	ignore the learned state, write nothing
the hand-tuned stock baseline	--fork off --n-cpu-moe N (or -ot)	keep the first layers' experts in VRAM, the rest on the CPU, as in upstream; the fair comparison for the cache

With --fork off the fork behaves as upstream apart from unrelated changes (the loader, MTP, merged upstream commits).

--moe settings (command line)

--moe takes comma separated name=value pairs, names are case-insensitive and _ equals -; A setting you give is used as given and never self-tuned.

key	default	meaning
slots=N	automatic: on when a MoE model does not fit in VRAM	expert slots per layer in VRAM; -1 sizes them from the free VRAM, a number fixes them, 0 = no cache (the rest of the fork stays on; --fork off is stock behaviour)
pf-slots=N	0 (off)	staging slots for host-to-GPU prefetch of whole expert tensors for big batches (3 is recommended when used, at most 4)
up-max=N	2	most expert uploads per layer and decode step; fewer means less PCIe and RAM traffic and a slower-changing cache
recent=N	64	how many recent tokens the cache looks at when it decides which experts to keep
pred-top=M	0 (off)	guess which experts the next layers will need and upload the likely ones early (the top M of the guessed ranking); experimental, see the evidence below
train-every=N	0 (the predictor is not trained while running)	train the predictor every N decoded tokens; implies pred-top=8 if pred-top is not given
autotune=0	autotune on	no self-tuning (same as -at off)
tag=NAME	none	research: put NAME into the file names of log=x (routing/<model>.<NAME>.<time>.routing); fork/run.py sets it to the test and prompt (game4-racing, edit4-2, long1k)
expert-owner=1	off	research, prompt batches only: the experts of a layer are uploaded to and computed on the card that owns the layer, over that card's own link, instead of all of them on the offload card. Meant for use with split=bw: each link then works on its own layers at the same time. Without a staging ring on the second card its uploads take the synchronous path, so judge it by the nsys timeline (x4 link share of the upload, stalls in the x16 upload stream) before relying on it.
split=bw	on by default with 2+ GPUs when the model does not fit in VRAM and is a MoE (split=mem: free-memory split)	layers by upload bandwidth: with two or more GPUs the layers are shared out in proportion to each card's pinned host-to-device upload rate (measured by the per-GPU probe at start), not its free memory. A prompt pass is bound by the expert transfer, compute is faster than the transfer, and a layer owned by a card on a slow link makes the offload card wait for it (x4 + x16: about 20% / 80%). A -ts of your own always wins. Decode prefers the memory split: Qwen3.8-Flash-Next game4c (2026-10-14, one run each) 46.5 decode t/s with this split (x4 card first, 15% of the layers, hit rate 92.5%) against 54.1 with 50/50 and the x16 card first (-dev CUDA1,CUDA0 -ts 50,50, hit rate 97.9%); a decode layout by memory is being worked on.
split-slow-scale	0.75	Factor on the bandwidth share of every card that is not on the fastest link when split=bw shares out the layers (the slow card also pays for the hand-offs of its layers). Measured calibration: GLM and MiMo on x16 + x4 are best at a 15% x4 share (bandwidth share 20% x 0.75); 20% costs MiMo 10% prompt speed
layer-heat=1|fit	off	research, crashes on the first prompt longer than the cache path (2026-10-14: CUDA invalid argument in the cross-card copy, being fixed): which layers sit on which card by how well their caches hit (the saved hot map: the share of a layer's uses its hottest half of the experts covers), the hottest on the weakest card, where a layer that swaps little loses least. Cards are ranked by capability = 1 / (link latency + one expert's upload + result latency of the probe). 1: each card keeps its layer count; fit: counts by memory, each weaker card takes the hottest layers that keep its misses within a budget scaled by its capability
swap-test=A:B	off	research: trades layers A and B between their cards at the first decode step (llama_context::swap_layers_fork: weights, KV and cached experts move through host memory, nothing is allocated)
pipe-stock=1	off	research: turn on the stock micro-batch pipeline across the cards although the experts are host-resident (tensor overrides normally switch it off). Needs -b larger than -ub; micro-batch k+1 then runs on one card while k runs on the other
split-order=fast	off (slowest link first)	research: in the multi-GPU prompt split (prefill-split) issue the fastest card's expert chain first instead of the slowest link's. With staged weights the slow link has nothing to stream ahead, and the first chain held the second back (per layer: x4 chain 30 ms alone, then x16 chain 111 ms)
split-slow-share=F	off (alpha x link share)	research: in the multi-GPU prompt split (prefill-split) the cards not on the fastest link hold F of the experts together (0.25: a quarter is uploaded to and computed on the x4 card). Without it the x4 share is at most 20% (alpha 1) and ~10% at alpha 0.5 (nsys, GLM 8k: the x4 card received 11% of the uploaded bytes)
split-chunks=N	1	research: in the multi-GPU prompt split every card's expert range is cut into N blocks, each staged, uploaded and computed on its own, so a card computes in step with its transfer inside the layer (nsys, GLM 8k: x16 idle while a 1.8 GB tensor arrived)
sim-moe-repeat=N, sim-dense-repeat=N	1	diagnostics: run prompt-sized expert matmuls / dense matmuls and attention N times in the CUDA backend (same result), so a small model gets a big model's compute-to-upload ratio in scheduler tests (OLMoE: moe 1, dense 3 is close to GLM 8k: compute 2.5x the upload, dense 2.5x the experts). Env: GGML_CUDA_MMID_REPEAT, GGML_CUDA_DENSE_REPEAT
pin-budget=BYTES	set by the server	load policy, not a user knob: with the automatic load mode, a model that does not fit pinned RAM is pinned as far as it reaches and the rest is mapped (never the whole model through mmap): the server sets this to the free huge-page pool plus the available RAM minus 4 GiB, the loader keeps the pinned host buffer for that many bytes of tensors in tensor order and maps the rest, and the log says weights pinned as far as they reach ... the rest mapped. A model that fits is pinned whole as before.
log=LETTERS	off (research)	every research and diagnostic log behind one switch, nothing is written unless you ask (--moe log=esr): e appends one line model,embhex,hots per run to embhot.csv in the working directory (the last layer embedding of the last token, 1 byte per float as hex, and this run's hot experts per layer; logdir=DIR moves it); s (or i) saves the model's own section of the state file at exit as state-snapshots/<date>_<HHMMSS>-<model>.ini (logdir=DIR moves it); g logs the GPU identity of the placement record; t writes the per-layer timing of every prompt batch (more than one token) as prefill-timing.csv (file,pass,n_tokens,layer,backend,dt_us,cum_us: how long each layer took on each card, including upload waits and the hand-off from the previous card; the scheduler synchronises at every layer, so the totals run somewhat above an undisturbed pass, the shares are what counts); p prints cache stats every 64 steps (p500: every 500); c writes the decode-step trace as moe-trace*; l (every generated token) or l64 (every 64th) appends each layer's output embedding, router input and selected expert ids to layertrace.csv (logdir=DIR moves it); x (or x64) appends only the selected expert ids of every layer to routing/<model file>.<YYYYmmdd-HHMMSS>.routing (binary, one file per prompt, --moe tag=NAME puts the test and prompt into the name: a header "MOER", version, layer count, k and the layer numbers, then per token an int32 position and layers x k uint16 expert ids; fork/sim/routing_csv.py converts it); r is read by fork/run.py (run history to csv and sqlite); details in research. The routed expert ids of every MoE layer are a backend diagnostic, GGML_MOE_LOG=FILE (see below). Nothing is read or written while running, so it costs no speed; preloading and seeding are unchanged

Names. The settings were renamed to say what they do. The old names (cache, inserts, window, prefetch-slots, predict, train, the knobs MARGIN, GATE, SWAP_FRAC, BIG, ... and their LLAMA_MOE_CACHE_<OLD> variables) no longer exist: an old --moe name is ignored, and an old state file entry is skipped, so the tuner relearns it. Case and - / _ do not matter in the new names. --moe-expert-cache N is unchanged and works on every build.

new name	old name		new name	old name
slots	cache		pred	PREDICT
up-max	inserts		pred-iso	STREAM
recent	window		pred-n	STREAM_M
pf-slots	prefetch-slots		pred-lead	OFFSET
pred-top	predict		pred-slow	STREAM_SLOW
train-every	train		pred-tune	SELF_TUNE
swap-lead	MARGIN		pred-auto	AUTO
swap-lead-per-link	LINK		pred-keep	SLOTKEEP
swaps-per-step	BUDGET		pred-slots	STREAM_SLOTS (env only)
vram-reserve-mb	MARGIN_MB		idle-up, idle-up-n	TBP, TBP_LAYERS
upload-wait	GATE		ev-cld	BIG
upload-wait-max-us	GATE_MAX_US		pin-hot	HOT_FRAC
upload-chunk-kb	CHUNK_KB		slow-min-stay	SLOW_STAY
upload-share	SWAP_FRAC		stay-bonus	STICKY
upload-now	JIT		l3-pf, l3-auto	L3PF, AUTO_L3
cpu-ram-gbs, ram-ceiling-gbs	CPU_GBS, DDR_GBS		trace-n, trace-skip	TRACE, TRACE_AFTER
wait-swaps	WAIT		pool	(new)

Any other name is a tuning knob of the cache engine (--moe upload-wait=3,swap-lead=0). A knob you set is never self-tuned; a name the
engine does not know is logged as unknown key and ignored. Values are numbers; 0/1 are off/on.

The cache keeps some experts in VRAM (the GPU computes those); any other expert is computed by the CPU or uploaded over PCIe first.
The knobs trade CPU work against uploads. Every example below is a complete option: llama-server -m model.gguf --moe <example>.

Evidence labels: proven = a paired measurement showed a gain (where and how much is given); unproven, on the way out = no
measured gain, will be removed or folded into the tuner unless a test shows one; no A/B = never tested on its own, kept as a
measurement input or a safety limit. Numbers are decode tokens/s unless stated; "chat" = short chat prompts.

The three knobs that matter most, in plain words

Each decode step a layer needs a few experts. The cached ones run on the GPU. For a missed one the engine can let the CPU compute it, or upload it into a
cache slot so it is a hit next time. An upload costs PCIe time, RAM bandwidth (it reads the same RAM the CPU is reading) and an eviction (a cached expert
has to make room). The three knobs answer three questions about uploads:

• swap-lead (which swaps are worth it). A missed expert only gets a slot if it beats the weakest cached expert by this many recent uses (over the last 64 tokens).
  0: any expert used more than the weakest cached one gets in; the cache adapts fast and churns more. A high value: a stable cache that can go stale.
  Default -1 is 0; -2 computes it from speeds (about the CPU read rate over the link rate: around 2 on an x16 link, 9 to 11 on a x4 link).
  Measured on GLM (2 x 3090): the speed-derived margin capped the hit rate; margin 0 gave 71.2% against about 65% and 21.3 against 19.9 t/s (+6.7%).
  Short chats like margin 0, long prompts did better with the speed rule, so the tuner chooses.
• upload-wait (when an upload may run). The CPU computing its experts and an upload both read system RAM and together reach its limit (on machine A about 38 GB/s for
  the CPU alone, 44 with a DMA running). An upload that overlaps the CPU's reading slows the CPU, which is the critical path. upload-wait=3 makes uploads wait, up to a limit,
  while the CPU computes, so they run in the gaps; upload-wait=0 uploads any time. Measured: +6% on a short GLM chat (22.10 against 20.87 t/s). upload-wait=2 starved the uploads: never use it.
• upload-share (how much). The most time uploads may take per step, as a share of the token time (0.25 is a quarter). More uploads help the hit rate but risk stalling the step.
  Measured: no benefit as a user setting (5 against the default gave the same hit rate, 64.6% against 64.7%; 0.5 with margin 0 was slower), so it is labelled "unproven, on the way out".

The tuner sets all three while it runs. If you change one, change one at a time and use --moe state=0, or the learned settings will mix into the comparison.

Swap decisions

knob	default	what it does	example	evidence
swap-lead	-1	An upload costs link time and only pays back if the expert is used again, so a missed expert must beat the one it evicts by this many recent uses. -1 = 0 (swap whenever it scores higher), -2 = from speeds (CPU read rate / link rate is about the uses an upload needs to pay for itself), n = fixed	swap-lead=0 swaps eagerly; swap-lead=2 swaps rarely	Proven on GLM-5.3-Flash 3.0-bit, 2x3090: the speed-derived margin (2 on x16, 9-11 on x4) capped the hit rate; swap-lead=0 gave 71.2% hit and 21.26 vs 19.93 t/s (+6.7%, 8 topic chats). Not universal: margin 0 wins short chats, the speed rule wins a long prompt, so the tuner picks
swaps-per-step	-1	Fixed swaps per step; -1 derives it from the measured upload time	swaps-per-step=4	No A/B
upload-share	0.25, tuned live	Uploads may take at most this share of a token's time; uploads that outlast the token stall it	upload-share=0.5	Unproven, on the way out as a user knob: GLM 2x3090, 5 vs default gave the same hit rate (64.6% vs 64.7%); swap-lead=0 with upload-share=0.5 was slower (20.62 vs 21.26 t/s)
swap-lead-per-link	1	Margin per upload link, so a GPU on a slow x4 slot gets a higher margin than one on x16; 0 uses one margin from the average upload	swap-lead-per-link=0	No A/B
upload-now	1	When a layer's router ids are known, upload some of the missed experts right then, so the GPU computes them this token while the CPU does the rest	upload-now=0 turns it off	Unproven, on the way out: GLM 2x3090, a first +4.5% was noise on a 6-round rerun (off 21.80, on 22.03 / 22.25, sd about 1); without admission it churns a cold cache (18.99 vs 20.70). W10 Qwen3.6: no change
lead	1	The predicted layers per link are the nearest ones whose upload still lands in time: the window starts at the upload time per expert on that link over the layer time. 0: every lookahead up to the predictor's count (the old behaviour)	lead=0
lead-span	1	Layers beyond the nearest feasible one that are also used	lead-span=2
admit	2	A missed expert may take a cache slot only after this many uses in the last 64 tokens (0: any miss). Higher values mean fewer swaps but a slower-adapting cache: of the experts used in a 64-token window 69% reach 2 uses, 52% reach 3 and 40% reach 4, and an expert admitted on its 3rd use brings only about 16% more further hits than one admitted on its 2nd. On Qwen Next admit 2 and 3 measured equal; 0 to 2 gave +3.4% decode on Qwen3.6	admit=3
admit-boot, admit-boot-steps	6, 128	While the cache is still cold (the first admit-boot-steps decode steps of a process that has seen no big prompt batch) a missed expert needs this many uses instead of admit: a strict entry saves uploads that cost more than they return. A big prompt batch fills the cache by adoption and ends this phase at once. admit-boot=0 (or any value <= admit) turns it off	admit-boot=0	Qwen3.6 game4, 100 tokens: admit 8 at the start +5.8% over 2 with the same hit rate; gone at 1024 tokens. The first version cost GLM 12k-prompt decode 5% until the prompt batch ended the phase
admit-slow	0	The same rule for layers on the slower link, where an upload costs 4-7x more (0: same as admit)	admit-slow=4
admit-jit	0	The same rule for upload-now uploads (0: any miss of this token may be uploaded)	admit-jit=1

Not fighting for RAM bandwidth

knob	default	what it does	example	evidence
upload-wait	3	The CPU reading experts and the upload DMA both read system RAM and together hit its limit. 3: an upload waits while the CPU computes uncached experts; 0 off	upload-wait=0 uploads at any time	Proven on GLM 3.0-bit 2x3090 short chat: upload-wait=3 22.10 vs 20.87 t/s (+6%, n=2); ties on long prompts. upload-wait=2 starved uploads (19.24 t/s): never use it
upload-wait-max-us	-1	Longest wait per tensor copy, microseconds; -1 = the measured mean layer time	upload-wait-max-us=800	No A/B
upload-chunk-kb	-1	With upload-wait=3: copy in chunks of this size and recheck the RAM budget before each; 0 = the whole tensor	upload-chunk-kb=512	No A/B
cpu-ram-gbs, ram-ceiling-gbs	38, 44	Assumed RAM read rate in GB/s of the CPU alone, and of CPU plus uploads together; starting values until the start-up probe measures them, ram-ceiling-gbs is the ceiling for upload-wait=3	cpu-ram-gbs=22,ram-ceiling-gbs=28 for a slower RAM kit	Measurement inputs, not tunables (PC1 DDR4-3200: CPU alone 37-38 GB/s, with both DMAs 44-45)
wait-swaps	1	The step waits for queued swaps to finish; 0 never waits and finished uploads appear at the next split	wait-swaps=0	No A/B recorded

Predictor streaming (uploads ahead of need). The whole group is unproven, on the way out unless the pending corrected A/B shows a
win: on GLM 2x3090 the predictor raised the hit rate (chat 46.6 to 56.1%) but not tokens/s (14.2 vs 13.9), and our PC1 experiment
lost 3-10% (hit rate +0.4-0.7 points) because uploads arrived too late and compete with the CPU for RAM bandwidth. On the W10 laptop
(RTX 4060, Qwen3.6-35B) an earlier +27-34% shrank to 0-3% once the AVX2 CPU kernels landed. Off by default (needs pred-top=M or train-every=N).

knob	default	what it does	example	evidence
pred	0	The learned predictors run at all; off by default and never switched on by the tuner (they exist only with --moe pred-top=M or train-every=N)	pred=1,pred-top=8,train-every=64	see above
pred-iso	on	Predicted uploads go into separate stream slots, so a wrong guess evicts nothing useful; 0 = they evict cache slots (old path)	pred-iso=0	see above
pred-n	12	Predicted candidates uploaded per target layer (over-guesses on purpose, no confidence cut)	pred-n=6	W10: 6 was best; offline about 80% of the uploads at 12 are wasted
pred-lead	1	Predict only layers far enough ahead that the upload lands in time on their link; a slow link needs more lead	pred-lead=0	see above; the dynamic-lead version is still being tested
pred-slow	1	Also stream onto layers served by a slower link	pred-slow=0	see above
pred-tune	1	The tuner tries one streaming knob at a time on real token times and keeps the faster setting	pred-tune=0	It turns the predictor off where it does not pay (PC1 GLM)
pred-auto	1	1: adjust lead and pred-n per link from the measured late share and precision; 2: uploads per target layer = time until that layer / measured upload time per expert; 0 off	pred-auto=2	pred-auto=1 +22% on W10, -3% on PC1 (before the AVX kernels)

Sizing

knob	default	what it does	example	evidence
vram-reserve-mb	0 (= 384, or from the batch size by autotune)	VRAM in MiB kept free per GPU after the cache takes its slots, so a longer context or batch does not run out of memory (not the same as swap-lead)	vram-reserve-mb=1024	Safety limit: measured post-load growth was 54-170 MiB per GPU

Experimental knobs (off by default; unproven, on the way out: none beat the default in our A/B tests, kept only for
experiments and probably removed in a later release; do not rely on them):

knob	default	what it does	example	evidence
ev-cld	0	An expert may only evict one with at most its own lifetime use count	ev-cld=1	Never won; off and not tuned
pin-hot	0	Pin the always-hot set of a layer on slower links	pin-hot=0.5	Never won in A/B
slow-min-stay	0	Minimum stay in steps of an expert in a slow-link layer's tier	slow-min-stay=64	Never won in A/B
stay-bonus	0	Bonus for staying in the cache	stay-bonus=1	+1.6% on one GLM chat run, within noise
pred-keep	0	A prediction may only replace a stream slot holding a lower-scored expert of this step	pred-keep=1	No data
idle-up, idle-up-n	0, 6	After a token ends, stream up to idle-up of its misses into the first idle-up-n layers while the output head and sampling keep RAM idle	idle-up=2,idle-up-n=6	No data
l3-pf, l3-auto	0, 0	Prefetch l3-pf experts per layer into the CPU's L3 cache (needs GGML_MOE_CCX_SPLIT and pinned threads); l3-auto=1 tests it live	l3-pf=2,l3-auto=1	The static per-L3 row split it needs alone lost 19% on GLM (Ryzen 3700X); pinning itself was neutral (21.73 vs 21.53)
trace-n, trace-skip	0, 0	Debug: record routing for this many steps, starting after this many	trace-n=2000,trace-skip=500	Debug only

More --moe settings

Every setting below is a --moe key (--moe policy=window,fixed=1); there are no environment variables for them. Most exist for experiments; the default is what we ship.

key	default	what it does
state=PATH	~/.cache/llama.cpp/moe-state.ini	path of the state file; 0 ignores it and never writes it
mode=stock|cache|retest	measured	stock or cache forces the placement; retest forgets the saved decision and measures again
fixed=1	0	deterministic mode, for perplexity and A/B runs that must repeat: no tuning, a fixed swap budget and swap-lead 4, and each step publishes exactly the uploads the previous step scheduled, so the cache contents (and with them which experts the GPU or the CPU computes) no longer depend on upload timing. Add upload-now=0: the just-in-time uploads size themselves from live link and CPU timing
tuned=...	state file	start from this tuner result instead of the saved one (0: none, every run starts cold)
ctl=FILE	off	a control file whose settings are re-read while running
pinr=N	0 (off for now)	a model that does not fit the RAM keeps its weights mmap'd; pinned copies of the slices of the experts that are uploaded to the GPU a second time (evicted and wanted again) are made as long as N/10 of the RAM lasts (pinr=3: 30%, pinr=0: off). The next upload of such an expert reads the pinned copy (direct DMA, about 2.2x the speed of pageable memory on a x16 link), cold experts stay in the page cache. Stops early when less than 6 GiB of RAM is available. A model that fits the RAM is pinned whole as before (-lm mmap opts out)
policy=add|window|hybrid|halve|lru	add	eviction score: add (recent use plus lifetime share), window (uses in the last 64 tokens only), lru (plain least recently used: the score is the time of the last use, no lifetime counts, no pay-back margin). Pure LRU: --moe policy=lru,admit=0 (admit is the default 2: a missed expert takes a slot only after 2 uses in the last 64 tokens)
global-weight=N	16	weight of the lifetime share in policy add
rel-decay=F	0.5	research, policy rel only: each step the relevance score of an expert is multiplied by F before the router probability of this step is added
halve-every=N	64	counts halve every N steps, so recent use dominates old use
adopt=F	1	share of each layer's cache slots that may keep experts the prompt pass already put on the GPU (1 all, 0.0625 a sixteenth; measured on GLM 12k prompt: 1/16 +0.3 hit points, 1/4 +3.8, all +11.4)
probe=0	on	skips the start-up probe that measures CPU read rate and each upload link
cpu-ram-gbs=N, ram-ceiling-gbs=N	probe	override the probed RAM rates in GB/s (same as the knobs)
max-batch=N	31	largest batch that still uses the cache path; bigger batches take the prompt path
prefill-d2d=0	on	stops prompt processing from copying experts between GPUs
prefill-split=F	tuned	how prompt batches spread experts over GPUs: 1 by link bandwidth, 0 the fastest GPU only, between = a mix; unset lets the tuner measure it
sync=0	1	stops the cache from waiting for the GPU between graph runs (unsafe, for experiments)
bounce-kb=KB (or bounce=MB)	0 (off; 256 recommended where RAM bandwidth is free)	Uploads from mmap'd weights are copied (memcpy) into the upload worker's pinned staging buffer in chunks of this many KB, then DMA'd from there instead of straight from the pageable mmap pages. Halves the time of an expert upload on Qwen Next (0.60 to 0.36 ms fast link, 0.95 to 0.69 slow link); decode +3% at best (within noise). Chunks of 256 to 512 KB give the full gain, 64 KB almost none, and larger sizes no more. Not used when the weights are pinned (direct DMA). Off by default: on GLM 3.0 (117 GB, many uploads) 256 KB cost 16% (16.4 vs 19.4 t/s), the memcpy competes with the CPU experts for RAM bandwidth
upload-threads=N	auto	upload worker threads; there is always one worker set per link, so a slow x4 copy does not block the fast link
pread=1	0	uploads with pread through a pinned buffer instead of copying from the mapped memory (copying from the mapping measured faster, so off)
drop=1, drop-stay=N	0, 1024	for models bigger than RAM: drop the RAM pages of VRAM-cached experts after they stayed cached this many steps (unproven, on the way out: -9% on MiMo with one GPU, off)
jit-pool=N, jit-pool-all=1	0, 0	extra slots for just-in-time uploads of slower-link layers; unproven, on the way out: costs 10-12% on GLM with two 3090s even when empty, off
neg-ids=...		diagnostic for negative expert ids
thread-tune=0	on	turns the thread-count tuner off
batch-tune=1	off	the prompt-batch thread tuner (needs 15 full batches to decide, so it never finished on prompts under about 16k tokens)
auto-place=1	0	places threads on physical cores per L3 domain
load-threads=N	auto	threads that read the model
l3pf-cpus=LIST	off	cores for the L3 prefetch (experimental)
defer=N, defer-mass=F	0, 1.0	experimental expert deferral (compute N selected experts during the next layer's attention). Faster (+8 to 12% on GLM) but changes the output (KL cost), so opt-in; not proven acceptable, on the way out
observe=0	on	no hot-expert counting on stock placement starts. The counting learns which experts the answer uses (decode batches of at most 8 tokens; a prompt batch routes differently and is not counted), so a later cache run starts from a warm map. It is skipped when every expert already sits on a GPU (the cache never runs there)
count-prompt=0	1	cache mode: prompt batches are counted in the recent-use window (they warm the cache before decode) but not in the lifetime counts that score eviction and become the saved hot map; the question's experts are not the answer's. Experiment: default stays 1 until an A/B on 12k-prompt decode shows a gain
observe-steps=N	512	the counting stops after N decode calls and the run goes on at full speed (0: never stop)
spec-depth=N		pins the speculative draft depth of the server
log=LETTERS, logdir=DIR	off, .	the research logs and where they go, see the log row above

Predictor settings

With --moe pred-top=M (or train-every=N) a small predictor guesses which experts the next layers will need.

key	default	what it does
predict-ahead=N	2 (0..8)	how many layers ahead it predicts
predict-max=N	2	most predicted uploads queued per layer on the non-stream path
predict-margin=F	0.1	a candidate is uploaded only if its score leads the M-th prediction by this much
predict-mu=F	0.5	step size of the online predictor
predict-stride=N	1	share one predictor over this many consecutive layers
predict-q8=0	on	keeps predictor weights in fp16 instead of 8 bit
predict-miss=F	0.05	predictors whose miss share stays above this are switched off
predict-file=PATH	auto	where the learned predictor is saved between runs; 0 = not saved
predict-ra=...		diagnostic

Backend diagnostics (environment variables, as in upstream's backends)

The GPU and CPU backends have no access to --moe; these stay environment variables, as GGML_* ones are in upstream: GGML_CUDA_HUGEFS (the pool, see below), GGML_CUDA_THP_SHARE and GGML_CUDA_REG_CHUNK_MB (pinning), GGML_CUDA_NO_GATE_FUSE (no fused gate kernel), GGML_SCHED_D2H_ASYNC=0 (synchronous device-to-host copies), GGML_SCHED_PROF (scheduler timeline), GGML_MOE_LOG=FILE (routed expert ids of every MoE layer, big), GGML_MOE_CCX_SPLIT (expert rows per L3 domain).

Faster loading of pinned weights: huge pages, the pool and the cache

With --load-mode pin (the llama-server default when the model fits in free RAM) the weights are read into pinned host memory so the GPU
can copy them directly. Pinning 100 GiB is slow with ordinary 4 KiB pages. The loader has three levels, tried in this order; each falls through
to the next without an error, so nothing here is required.

level	needs	what it does
1. Cache file (GGML_CUDA_HUGEFS)	the pool and the mount below, set up once with root	the weights of a model are kept in a file in RAM that outlives the process; a later start maps it and skips the disk read
2. Reserved pool pages	the pool below	the buffer is placed on free 1 GiB (then 2 MiB) huge pages of the pool, no page-by-page work
3. Hybrid (default, no setup)	nothing	the part of the buffer the kernel can back with 2 MiB transparent huge pages (70% of the free 2 MiB blocks in /proc/buddyinfo) uses them, the rest 4 KiB pages; one cudaHostRegister call over everything

Measured (GLM-5.3-Flash 3.0-bit, 109.4 GiB, --load-mode pin, cold page cache, machine A: Ryzen 7 3700X, 125 GB DDR4-3200, 2x RTX 3090, kernel 7.1.5, driver 580.173.02)

comparison	time to ready	runs
plain 4 KiB pinning (16 read threads)	91.3 s median (87.8 to 92.7)	ABBA, n=4 each
hybrid (level 3)	72.8 s (69.4 to 75.9), the pin step 60.5 s down to 44.1 s	same runs
warm cache file (level 1) against the pool path (level 2), one-chunk perplexity run, --moe state=0	34.5 s (34.3 to 35.7) against 56.6 s (55.6 to 60.8); the whole run 41.1 s against 62.9 s	ABBA, n=4 each

The perplexity was identical in every run of the cache comparison (3.7227), and a cold fill, two warm loads and a no-cache load on a
longer text all gave the same value (6.6026): the cache returns the same weights. One model, one machine, one kernel and driver: how much you gain elsewhere
depends on how fragmented your free memory is and how fast your RAM is. The first load of a model still reads the disk (about 47 s here); only
later starts are fast. Loading the pinned weights also holds that much RAM for as long as the process runs.

One-time setup: resident_pin.sh

Linux only (Windows uses the old path). The kernel needs 1 GiB huge pages (CONFIG_CONTIG_ALLOC, the CPU flag pdpe1gb); tested on kernel 7.1.5 only.

sudo fork/resident_pin.sh mount              # the most the machine can spare: total RAM minus a margin (default max(24 GiB, 20% of RAM), HUGEFS_MARGIN_GIB=N to change)
sudo fork/resident_pin.sh 100g               # exactly that many GiB (a number with G), or --pages N, or the path of a model file to size it for that model
sudo fork/resident_pin.sh unmount            # delete the cached models, unmount, give the pages back

It reserves the 1 GiB pages (it drops the page cache and compacts memory first so nothing has to be migrated), and mounts a hugetlbfs at
/mnt/huge1g owned by you. Root is only needed to change the pool or the mount; with a big enough pool and the mount in place it runs as a normal user.
The pool stays reserved until unmount (or a reboot), so that RAM is not available to other programs meanwhile: this is a setup for a machine
that serves one model, not for a desktop. Reserve once; the loader never grows or shrinks it. For a model that only partly sits in host memory
(layers on the GPUs) the cache holds just that part, so give the size by hand if the model's file size exceeds the limit.

Then start the fork with the cache on:

GGML_CUDA_HUGEFS=/mnt/huge1g llama-server -m model.gguf --load-mode pin

The first load of a model creates <key>.w.part, fills it while reading the weights and renames it to <key>.w when complete. The key is made
from the model file (device, inode, size, modification time) and the layout of the tensors in the buffer, so a changed file or a different
split of layers between GPU and RAM gets its own cache file. When the pool is full, the next fill deletes the least recently used cache file
that no process has mapped (a running process holds a lock on its file, so a model in use is never evicted); if there is still no room it
silently uses level 2 or 3.

Do not boot with hugetlb_cma=

Huge pages taken from a CMA area cannot be pinned for the GPU: the NVIDIA driver pins with pin_user_pages(FOLL_LONGTERM), the kernel refuses
that for CMA pages unless it can migrate them, and a 1 GiB page inside the area has nowhere to go. cudaHostRegister then fails with
invalid argument (we measured this for every flag and chunk size). resident_pin.sh mount refuses to run if that option is on the kernel command
line. Allocate the pool at run time, as the script does, from ordinary memory.

Environment variables for loading

variable	default	what it does
GGML_CUDA_HUGEFS	off	mount point of the cache (/mnt/huge1g); unset or without a mount the loader uses level 2 or 3
GGML_CUDA_THP_SHARE	0.7	share of the free 2 MiB blocks used for transparent huge pages in level 3. 0.9 was faster (70 s) in five runs but once froze a load for more than 13 minutes while memory was heavily used; 0 is plain 4 KiB pages
GGML_CUDA_REG_CHUNK_MB	whole buffer	register the buffer in chunks of this many MiB. Do not use: a tensor that straddles two registrations makes cudaMemcpyAsync fail with invalid argument at the first prompt batch (measured: perplexity crashed with 4 GiB chunks, ran with one registration)
GGML_CUDA_NO_PINNED	off	never pin the weights
GGML_CUDA_NO_REGISTERED	off	pin big weight buffers with plain cudaMallocHost instead of registering the loaded pages (diagnostic)
LLAMA_LOAD_THREADS	half the CPU threads, at most 16	threads that read the model file into the buffer; 1 keeps the sequential reader

If the pin fails: no silent mmap fallback

With --load-mode pin (the default when the model fits in RAM) a failed cudaHostRegister is reported and never replaced by an mmap load, because that would only make the run slow without saying why.
The log says weights NOT pinned: mmap / page cache when the weights are mmap'd (a model that does not fit, or --load-mode mmap) and pinned otherwise.
If the registration stalls or fails, the message says that a reboot defragments memory; the process exits after the third failure (exit code 75 on a stall). fork/run.py retries the start three times.
Every run also writes its effective settings (autotune result, admit, load mode, every knob) as the run.knobs line of the state file.

If something goes wrong

• only N of M pages could be allocated: memory is too fragmented or too full; stop other programs, run resident_pin.sh unmount, mount again with a smaller size, or reboot.
• No hugetlbfs cache line in the log: the mount is missing, the pool is too small for the file, or another process is filling the same model; the load used level 2 or 3.
• A load that stalls for minutes while the machine is swapping: lower GGML_CUDA_THP_SHARE (or set it to 0), and do not load a second large model while a cache file holds most of the RAM.

The state file

~/.cache/llama.cpp/moe-state.ini (--moe state=PATH moves it, --moe state=0 ignores it and never writes) holds one
[model name + size] section per model:

line	content
hot.N = c0 c1 ...	lifetime use count of every expert of layer N. At start the most used experts are loaded into the cache first, so the first prompt is already warm
tuned.lK = NAME=value ...	what the self-tuner settled on, for K upload links (GPUs)
place.g<gpus>.<GPU hash>.v3.cache, .stock, .decided	measured prompt and decode speed of cache and stock placement for this GPU set, and which one won
place.g<gpus>.<GPU hash>.v3.link<k>	the link bandwidth (GB/s, rounded to half octaves) seen when the placement was measured; a later probe that differs by a factor of 2 or more drops this hardware's placement records
place.g<gpus>.<GPU hash>.v3.pp_c_us	compute of one prompt token on the offload card, all layers, in microseconds (measured on one-card passes): a new start plans its first prompt from it (pass sizes, and whether a second card pays for the split) instead of running it on one card to measure

Counts only seed the start; during a run the cache scores by recent use. It is plain text, safe to edit or delete.

Prompt processing: passes, staging and the second GPU

A prompt pass walks every layer over all its tokens. The experts of a layer stream from RAM into a ring of staging slots on the card, ahead of the
compute; the dense weights (attention, router, norms) stay on the cards. Every pass streams every expert the card does not hold, so the number of
passes, what the pool lends to the pass and which cards take part are calculated per prompt, then corrected by what the passes measure.

step	decides	from
pass planner	passes per prompt and staging slots per pass	compute per prompt token (measured on every one-card pass), the upload of one pass (expert bytes the pool does not hold over the link rate), what the pool can lend
split planner	whether a second card pays for a pass of this size; the helper's share of the tokens; per layer its share of the experts	the same compute and link rates, each card's held experts, the hand-offs between the cards
measured choice	one card or split, when the plan puts the two closer than 15%	prompt speed of both on real passes of this size
split tuner	the final shares for this pass size	prompt speed of the split passes: one share at a time, the step halves when it gets slower, at most 12 passes, then it stays

The split is on by default (dense-split=auto, switch it off with dense-split=0): its copies of the dense weights live in expert-pool blocks that are lent only to prompt passes that split (mirror-pool), so decode keeps the full cache; a card with no pool, or whose copies would take more than half of it, keeps them in a buffer of its own.

key	default	meaning
dense-split=auto|1|0|one	auto	the dense part of every layer runs on all cards at once, each card on its own token range. Needs a copy of every layer's dense weights on every card, made once at load out of that card's expert pool (Qwen3.8-Flash-Next: 3.6 GB onto the x4 card, 0.6 GB onto the x16; GLM-5.3-Flash: 6.8 GB and 1.5 GB). auto: the split planner and the measured choice decide per pass size. 1: every pass of dense-split-min tokens or more is split. one: the copies are made, every pass runs on one card (tests). Token-range graphs exist for GLM-5.3, Qwen3-Next, Qwen3.8-Flash-Next, MiMo-V2, Qwen3.5 MoE and DeepSeek-4 (compiles, not run: no model here); other models get the expert split alone
dense-split-min=N	512	the smallest pass that is split
dense-split-share=F	planned	the helper cards' share of the tokens, fixed by hand (tests). Unset: the split planner's share, then the tuner's; 0.55 when there is no plan
split-plan=0	on	no split planner: the measured choice alone (one card first, then one split pass, then the faster of the two) with the link-ratio start shares
split-tune=0	on	the planner's shares stay, nothing is tuned from measured passes
split-experts=auto|0|1	auto	whether the helper cards of a dense split also compute experts. auto: the split planner per layer and the tuner. 0: dense only, the helpers take token ranges of the dense part and the card with the fastest link computes every expert (Qwen3.8-Flash-Next 3k on x16 + x4: 1390 prompt tokens/s against 1237 with 21% of the experts on the x4 card, whose link then carries 41 ms of uploads per 49 ms layer). 1: experts by link ratio with the per-layer share controller, whatever the planner says
one-range=1	off	research, with dense-split=auto|one: a pass that runs on one card runs as one token range on the card with the fastest link, the dense part of every layer from that card's copies, so nothing is handed to the cards that own the other layers. Measured slower on x16 + x4 (Qwen3.8-Flash-Next 3k: 1091-1146 prompt tokens/s against 1132-1211), same perplexity
split-f16=0|in|1	0 (f32)	the precision of the hidden states that cross between the cards. 0 (default): f32; 1: f16, half the bytes on the helper's link (f32 was ~2% slower at 3k); in: f16 to the card that computes the experts, f32 back (~3% slower). Perplexity, Qwen3.8-Flash-Next, 10 chunks paired against one card: f16 +0.18% (IQ1_M) and +0.42% (IQ3_S); f32 +0.09% and +0.09%, within noise; f16 minus f32 on IQ3_S +0.33 +- 0.17%. f32 is the default since 2026-10-20
split-auto=0	on	stops the per-layer expert share controller of the expert split: the shares stay at their start
alias=0	on	the compute buffer of a graph is not lent from the expert pool but allocated beside it for the whole session (it then costs cache slots while decoding)
alias-all=0	on	a prompt pass borrows blocks of the biggest size class only (as before 2026-10-13). On: of every size class, so the largest pass no longer depends on which class is the biggest
stage-shrink=1	off	old behaviour: the micro-batch is halved until the graph and all pf-slots staging slots fit the pool. Off: the pass keeps its size and the ring takes the slots that fit, at least 3 (MiMo 4k: 8 passes of 512 tokens at 127 prompt tokens/s became one pass at 543)
pass-plan=0	on	no pass planner: equal passes up to -ub, pf-slots slots
pf-resident=0	on	every pass uploads every expert. On: the staging fill copies the experts the card's pool holds on the card and uploads the runs between them (Qwen3.8-Flash-Next: about 40% of the 49 GB of a pass)
pf-count=0	on	no counting of the experts a prompt batch uses on the GPU (the counts feed the cache's lifetime scores once per expert and prompt)
log=w		prints the pass planner's choice per prompt
reheat=0	on	after a prompt the blocks it borrowed come back empty; on: each is refilled at once with the best-scored expert not cached (background uploads that never delay a decode step and cost no swap budget). Off: they fill only as decode misses earn them
log=f		the pool per card before a prompt, right after it (its blocks back) and 8 / 32 / 128 decode steps later: blocks holding experts, uploading, free, lent, the card's free memory. fork/experiments/pool_fill.py LOG... turns the lines into a table

Known gap (2026-10-14): the dense-split copies are separate buffers made at load and kept for the whole session, so with the split on they cost expert
cache in decode too (the x4 card holds 3.6 GB of Qwen3.8-Flash-Next copies). Planned: the copies live in blocks lent for the prompt only.

What the server reports about its choices

Every automatic choice is printed at the default log level (-lv 3), and fork/run.py records it in run-history.csv and runs.db:

log line	when	run.py columns
layer layout: rule=bw CUDA0=8(0-7) CUDA1=40(8-47) output=CUDA1	load	layout (rule: ts your -ts, bw, plan, dec, mem free memory)
MoE placement: cache|stock (...), re-measuring stock this run	load	placement (stock (re-measure) when a cache start was given to a stock measurement)
MoE model ... ctx N, ubatch N	load	ubatch_max
prefill decision: prompt_tokens= passes= pass_tokens= staging_slots= mode= gpus= skipped= helper_token_share= helper_expert_share= prompt_tps= one_card_tps= split_tps= chosen_by="..."	every decode call with prompt passes	pf_mode, pf_gpus, pf_skipped, pf_tok_share, pf_exp_share, pf_passes, pf_pass, pf_slots, pf_one_tps, pf_split_tps, pf_by (largest prompt of the run), pf_all (every prompt)

mode: one-card (every layer's experts on the card with the fastest link; the other cards run only the dense part of their own layers), dense-split
(the dense part on all cards by token ranges, the experts on the fastest card), dense+expert-split, expert-split (decided at load), mixed.
chosen_by says why: an option forced it, the planner was sure (more than 15% apart) or close, the faster measured mode of that pass size, the slower one
measured again (every 16th pass), the other one measured once, the nearest measured size, the tuner's state, or the load-time expert split.

fork/run.py misfires [--days 7] [--tol 0.05] [--model M] lists runs whose prompt ran in a mode another run of the same model, machine and pass size
beat by more than tol (a second card skipped where the split was faster, or the reverse), and every stock re-measurement of the placement.

Limits: the split planner handles two GPUs (with more, the measured choice decides and the shares start from the link ratio); its split estimate was
21-26% short of the measured pass before calibration, one card within 4%.

Context and KV cache

With a MoE model whose experts stream from RAM, -np 1 and flash attention (the default auto), the KV lives in the expert pool (layout: img/vram-layout.svg). Nothing is reserved at start: each request claims the cells it needs from the pool (taking the coldest experts' blocks) and gives them back at its end.

setting	default	meaning
-c N	none: the model's trained maximum	caps one request (prompt + answer); reserves nothing in this mode
-n N / max_tokens	unlimited	bounds the answer; with -n given, -c is not applied
--moe kv-tail=0	on (1) when the experts stream from RAM	the normal KV cache: a static context, sized when no -c is given as the largest one whose KV and context-scaled buffers take at most 30% of the expert-cache VRAM (capped at the model maximum). Also used automatically with several slots (-np > 1), without flash attention, or with different K and V types
--moe kv-token-major=1	on with kv-tail	K and V of all layers of a card as one record per token (what lets the KV grow)
--moe kv-fallback=LIST	k8v8,k4v4,ram	the tiers a request may use after f16 when its KV does not fit the pool: k8v8 = q8_0 K and V in place, k4v4 = q4_0 in place, ram = whole layers' KV to host RAM (attention for those layers on the CPU; their VRAM goes back to the experts). Tried cheapest first. k8v8,ram skips q4_0 (the RAM tier then keeps q8_0); none = f16 only, a request that does not fit is refused with the numbers

A request picks its tier at its start from its length (prompt + max_tokens); a KV that fills up during the answer moves up a tier inside the same request. A later, shorter request converts the rows it keeps back down to f16 (layers moved to RAM stay there for now). q8_0 holds about 1.9x the tokens of f16, q4_0 about 3.6x.

Perplexity of quantized K and V, OLMoE-1B-7B IQ4_XS, CPU, 10 chunks (runs.db 15023, 15026, 15028, 15030, 15037): f16 2.1946, q8_0/q8_0 2.1926, q8_0 K + q4_0 V 2.1969, q4_0 K + q8_0 V 2.2016, q4_0/q4_0 2.2073 (+0.58%). K is the sensitive tensor; k8v8,ram keeps the KV at q8_0.

Several GPUs and MTP

• The cache is sized per GPU from its free VRAM and the experts spread over all GPUs; each GPU keeps its own slots and upload
  link. A card on a slow slot (x4) helps less, its uploads take 4 to 7 times longer, and prompt processing goes to the fastest
  link. -ts, -dev and -sm work as in stock; see multi-GPU usage.
• MTP works together with the cache: pass -md and --spec-type draft-mtp as in the MTP section above. The draft head is
  small and stays in VRAM, the cache serves the main model.