Experts only in VRAM (--vram-pin-gib, --second-gpu-pin-gib) by txiki739 · Pull Request #20 · eddoursul/Lamina

txiki739 · 2026-10-12T07:08:17Z

Builds on #19 (elastic K/V) and #13 (the arena load): review the last two commits.

What

The profile's hottest experts, up to G GiB, live for good in the main GPU's lowest cache slots and the next ones in the
second GPU's, and the RAM arena leaves them out: each GiB pinned is a GiB of RAM. They are read from the model straight
into their slots, the adaptive tiers never evict them nor take an expert the RAM lacks, the prompt path's loans and the
elastic K/V stay above them (startup refuses budgets that do not fit, saying by how much to lower them), and a layer
routing an expert only the second GPU holds always gives that GPU its share. Without the flags nothing changes. The
last commit holds review fixes (the PCIe share asks whether a layer is mapped, the prompt chunk's halving probe keeps
away from the pinned slots, --ple-io ram loads after the arena, the dequant identity test as a CMake target).

Why

A model whose experts outgrow the RAM: huihui's Q8_0 of Qwen3.8-Flash-Next has 119.5 GiB of experts; on a
125.7 GiB machine it runs with them pinned across both cards.
Faster even when everything fits: pinned experts are never lent to the prompt path, so after a long prompt the
hottest ones are still in VRAM. Two GPUs, six-prompt decode mean:

per card	UD-IQ4_XS	UD-Q4_K_XL	huihui UD-Q4_K_XL
none	118.2 tok/s	85.5	86.7
8 + 8 GiB	127.4 (+7.8%)	90.6 (+6.0%)	90.3 (+4.1%)
11 + 11 GiB	-	83.3 (too few slots left for the prompt path)	-

UD-Q4_K_XL right after an 18K document 75 -> 90 tok/s; the 18-step agent session 85.4 -> 89.1. See
README: two GPUs, experts pinned in VRAM.

Q8_0 and UD-Q6_K_XL on a 128 GB machine (what made these flags necessary)

huihui-ai's abliterated Qwen3.8-Flash-Next on the RTX 3090 + RTX 5060 Ti, our current settings for each (two runs
each for decode, three for UD-Q6_K_XL):

	UD-Q4_K_XL	UD-Q6_K_XL	Q8_0
routed experts	~72 GiB	101.7 GiB	119.5 GiB
experts only in VRAM, per card	-	6 + 6 GiB	11 + 11 GiB
decode, six-prompt mean	84.0 tok/s	55.9	41.8
decode, RTX 3090 alone	68.6	42.5	does not fit
18-step agent session	84.6	57.7	44.6
reading an 18K / 200K prompt	1,782 / 2,284 tok/s	1,457 / 2,109	1,330 / 2,057
KL from Q8_0 (x 1000) · same top-1, 19,208 teacher-forced tokens	64.1 · 91.2%	17.7 · 95.1%	21.2 · 94.8% (a second run)
perplexity over the same tokens	4.860	4.965	4.932
242 checks graded by code (Spanish, reasoning, tools, long context)	230	237	236

(huihui's UD-Q4_K_XL now runs with 8 + 8 GiB pinned too: 86.7 -> 90.3 tok/s.)

What the Q8_0 costs on 128 GB of RAM (125.7 GiB):

It does not load without the pins. 119.5 GiB of experts against ~120 GiB available with nothing else running.
With 11 + 11 GiB only in VRAM the engine takes ~97.5 GiB and ~19 GiB stay free (a prompt cache of 32 checkpoints and
10 GiB of parked conversations still fit; at least 6 GiB free once they are full).
One card is not enough. With only the 3090's pins ~108 GiB would have to fit in ~120: not runnable. And 11 + 11
is the ceiling: more pins leave the prompt path too few cache slots to borrow (with UD-Q4_K_XL at 11 + 11 an 18K
prompt already read at 532-1,078 tok/s instead of ~1,800).
No room for the PLE table in RAM (28.8 GB), so it keeps --ple-io direct (the Q4/IQ4 duals gain +33% on an 18K
Spanish prompt from ram).
Decode is bound by the RAM. Once both cards are full, ~1 GB of experts per token comes from the DDR4 (43 GB/s
measured): half UD-Q4_K_XL's speed whatever the prompt, and an accepted draft does not help (each extra token brings
its own experts). The two cards serve 87.8% of UD-Q4_K_XL's routed experts, 76.6% of UD-Q6_K_XL's and 68.1% of the
Q8_0's, and each miss is bigger: per token UD-Q6_K_XL reads ~2.9x and the Q8_0 ~4.4x what UD-Q4_K_XL reads from the
RAM. What read less from it won: --pcie-frac 0.25 (+5%), --spec-min-p 0.8 (+2.6%) and --feed-max 128 (the first
token of a short chat in 2.0 s instead of ~2.9).
The files cannot stay in the page cache next to a ~97 GiB arena, so every start reads the experts from the disk.
huihui's Q8_0 is a re-split GGUF: it needed the loader change of native dense: a re-split GGUF may name the architecture on every shard #15 (now in custom).
Route-resident routing (upstream 5df35dc, route-resident: one warp per token in the swap kernel Niko1221/Strata#1737) gives the Q8_0 +55% at margin 0.5, but costs as much quality as
going down to UD-Q4_K_XL (KL 0.024-0.038 against 0.006 between two plain runs), so we keep it off.

What we run instead: a UD-Q6_K_XL made from the Q8_0 (llama.cpp, unsloth's UD-Q6_K_XL recipe and imatrix; it keeps
the Q8_0's abliteration). It is as close to the Q8_0 as a second Q8_0 run is (KL 0.018 against 0.021) and 34% faster
(55.9 against 41.8 tok/s); its 101.7 GiB of experts fit in RAM, yet 6 + 6 GiB pinned still pays (54.2 -> 55.9) because
it reads less of the RAM. Details: README: a model whose experts do not fit in RAM.

Where this sits (measured today on your new `custom`)

custom at e3727cb against our engine (your previous custom + this series + the upstream pieces you have merged
too), our configs on a Ryzen 7 5700X + 128 GB DDR4 + RTX 3090 (+ RTX 5060 Ti 16 GB), Linux. custom runs the same
configs without the flags it lacks (--kv-grow, --adapt-decay, --vram-pin-gib / --second-gpu-pin-gib):

six-prompt decode mean, two interleaved runs each	`custom` `e3727cb`	our engine (this series)
UD-IQ4_XS, RTX 3090	80.6 tok/s	93.7 (+16%)
UD-Q4_K_XL, RTX 3090	64.4	69.4 (+8%)
UD-IQ4_XS, RTX 3090 + RTX 5060 Ti	103.8	126.1 (+21%)
UD-Q4_K_XL, RTX 3090 + RTX 5060 Ti	78.2	92.0 (+18%)
reading an 18K prompt, both cards (UD-IQ4_XS / UD-Q4_K_XL)	2,110 / 2,010-2,270 tok/s	2,530 / 2,420
the first chat right after READY, both cards (`--ple-io ram`)	67.8 / 52.8 tok/s	110.9 / 85.8

The per-change measurements and the configs: README: what the changes on top of eddoursul's custom gave,
settings and how to run, examples/r7-5700x.

This series (rebased onto `e3727cb`)

load_main: strata-load compiles again #12 load_main: strata-load compiles again
Linux startup: the expert arena filled with preadv and page-locked layer by layer #13 Linux startup: the expert arena filled with preadv and page-locked layer by layer
--ple-io ram: READY only once the PLE table is in RAM #14 --ple-io ram: READY only once the PLE table is in RAM
cpu pool: a native layer's gate/up and down rows as one batch (STRATA_POOL_FUSED=1, opt-in) #16 cpu pool: a native layer's gate/up and down rows as one batch (STRATA_POOL_FUSED=1, opt-in)
Elastic K/V (--kv-grow): the K/V takes VRAM as the context grows, the expert cache holds the rest #19 Elastic K/V (--kv-grow): the K/V takes VRAM as the context grows, the expert cache holds the rest
Experts only in VRAM (--vram-pin-gib, --second-gpu-pin-gib) #20 Experts only in VRAM (--vram-pin-gib, --second-gpu-pin-gib)
adaptive tier: --adapt-decay F (upstream's option), refused outside (0, 1) #21 adaptive tier: --adapt-decay F (upstream's option), refused outside (0, 1)

Closed as already in custom after the 0.1.43 merge: #15 (re-split GGUFs), #17 (upstream's CPU kernels), #18 (upstream fixes).

Tested on `custom` at `e3727cb`

Smoke runs (UD-IQ4_XS, greedy, a Spanish chat answer and a code prompt, RTX 3090 or both cards): both cards with 8 + 8 GiB pinned (with #19 and #13): 69.2 / 111.4 -> 110.9 / 131.9 tok/s; one card without the flags unchanged.

… expert cache holds the rest Upstream 0.1.40's elastic K/V (Niko1221/Strata 2fbe321 + 4da8a57) wired into this fork's engine. Since the 0.1.43 merge this tree has upstream's vmm.hpp/vmm.cpp and the expert cache's VMM arena (set_vmm, vmm_range, slot_offset); this adds the rest: the K/V pools in VMM ranges, the slots given up, and generate.cpp's, layer.cpp's and the prompt cache's parts. At --max-context 200192 the int8 K/V is 2.6 GiB allocated at start; on a 3090 that is 7,222 -> 6,142 expert slots and -14% decode (UD-IQ4_XS, the six prompts: 95.3 tok/s at 32K, 83.6 at 200K). With --kv-grow (or STRATA_KV_GROW=1) the K/V pools and the cache arena are CUDA VMM ranges: the pools map 2 MiB chunks for the cells the requests reach, and when a request needs more, the slots below the prompt path's loan give their experts up (a hotter one moves into the coldest loan slot first) and their chunks are mapped into the K/V. No address moves, so no graph is captured again, and --max-context cells always fit. This engine's parts: - layer.cpp: each state's arrays (kv_row_bytes, every K/V format) at chunk boundaries of its own range; the state arena no longer holds them; qsa_state_zero zeroes what is mapped. - the bound of the slots given up is the prompt path's loan for the longest chunk with a whole layer streamed: its buffers grow with the experts the cache does not hold (Prefill::bytes_for), so the loan at start was not a bound (a later loan reached the given-up slots: an illegal address in the next prompt). - the prompt cache grows the K/V before it restores a stashed sequence or stashes the live one (all their cells), and a trim never goes below the live sequence; a far shorter request trims only after two in a row (an agent's side requests between turns would otherwise shrink and regrow it every turn). - the adaptive tier drops given-up slots from its free lists; the residency uploads go through res_put. - --serve on one GPU only (no --second-gpu: its tier would need the ranges' access granted); when the cache cannot give slots up the whole K/V is mapped at start. Checked on a 3090 (UD-IQ4_XS, fixed cache of 3,000 slots, no adaptation, the K/V from 4,096 cells in steps of 2,048): grown into new VRAM, the four prompts' tokens are bit for bit the run without --kv-grow; grown by giving up 91 slots (83 moved), the prompt read while they were given up differs (those experts round on the CPU, as any miss), the rest are identical again after the trim refills them. Default unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… alone) The second GPU's tier opened its cache through the VMM arena as well, whose chunks belong to the main GPU (the device current at vmm.cpp's first call): 'second GPU 0 unused: mapping 14.57 GiB failed: out of memory'. The arena switch is cleared once the main cache is open. Nothing reads the main GPU's ranges from the other card (its rows go through the host), so --kv-grow keeps the second tier: UD-IQ4_XS on a 3090 + 5060 Ti, 107.7 -> 116.3 tok/s with STRATA_POOL_FUSED=1, both tiers full (7,173 + 6,446 slots). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

With --mmap-experts (UD-Q4_K_XL at 64 GB of RAM: its experts read from the NVMe) the engine ended in a crash after the first growth, on one GPU (a segfault in the next request) and on two. Upstream's elastic K/V asks for every expert in RAM (not the low-RAM tier) too; the K/V is then allocated whole, as without --kv-grow, and the log says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…yer by layer After he-be/Strata 32f8912 by masahiro hibi (pread on threads, the RAM copy registered after it), fitted to this engine's arena: cudaHostAlloc zeroed and locked all 55 GiB on one thread before the load (~16 s here). Now the arena is one anonymous range, every layer on 2 MiB-aligned pages of its own; the load's threads read each role tensor's slices with preadv straight into the experts' places (16 a call, no staging buffer and copy) and page-lock their layer with cudaHostRegister as soon as it is in place. From a refused registration on, the layers are locked resident, as the cudaHostAlloc path's tail. UD-IQ4_XS, RTX 3090: ready in 38-43 s from the disk (52-55 s before), 14-17 s with the GGUF in the page cache (33-50 s), both cards 22-24 s (50-53 s); with the RAM capped at 60 GB 38-40 s (55 s), no swap; the engine exits in 2 s (6 s). STRATA_LOAD_VERIFY=1 reads every expert again through std::ifstream and compares it with the arena: all 24,576 equal for UD-IQ4_XS, UD-Q4_K_XL and Huihui. The same tokens at a fixed cache (6 of 6 prompts, two rounds) and the same decode speed. STRATA_PIN_AFTER_COPY=0 keeps the cudaHostAlloc path, STRATA_LOAD_THREADS and STRATA_LOAD_BATCH size the reads. Windows unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…whose experts outgrow the RAM The profile's hottest experts, up to G GiB, live for good in the main GPU's lowest cache slots and the next ones (those the second GPU takes first) in the second GPU's, and the RAM arena leaves them out: each GiB pinned is a GiB of RAM (Huihui's Q8_0 Flash-Next, 119.5 GiB of experts, on a 125.7 GiB machine). They are read from the model straight into their slots, the adaptive tiers never evict them nor take an expert the RAM lacks, the prompt path's loans and the elastic K/V stay above them (startup refuses the budgets otherwise, saying by how much to lower them), and a layer routing an expert only the second GPU holds always gives that GPU its share. Without the flags nothing changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ed slots in every loan probe, the dequant test registered - expert_source: pcie_ok asked device_alias(layer, 0), null when expert 0 lives only in VRAM: such a layer lost its PCIe share. ExpertSource::layer_mapped. - the prompt chunk's halving probe keeps away from the pinned slots, as the loans do (it aborted instead of halving). - the elastic K/V is checked against the pins only when it is on; off, the existing fallback maps it whole as before. - --ple-io ram: the table's RAM load follows the arena, which loads late with pins. - the coalesced Q8_0 dequant's identity test is a CMake target (dequant_q8_0_identity), in English. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

txiki739 · 2026-10-12T12:30:06Z

For the record, the models that made these flags necessary, measured here:

huihui-ai's abliterated Qwen3.8-Flash-Next on the RTX 3090 + RTX 5060 Ti, our current settings for each (two runs
each for decode, three for UD-Q6_K_XL):

	UD-Q4_K_XL	UD-Q6_K_XL	Q8_0
routed experts	~72 GiB	101.7 GiB	119.5 GiB
experts only in VRAM, per card	-	6 + 6 GiB	11 + 11 GiB
decode, six-prompt mean	84.0 tok/s	55.9	41.8
decode, RTX 3090 alone	68.6	42.5	does not fit
18-step agent session	84.6	57.7	44.6
reading an 18K / 200K prompt	1,782 / 2,284 tok/s	1,457 / 2,109	1,330 / 2,057
KL from Q8_0 (x 1000) · same top-1, 19,208 teacher-forced tokens	64.1 · 91.2%	17.7 · 95.1%	21.2 · 94.8% (a second run)
perplexity over the same tokens	4.860	4.965	4.932
242 checks graded by code (Spanish, reasoning, tools, long context)	230	237	236

(huihui's UD-Q4_K_XL now runs with 8 + 8 GiB pinned too: 86.7 -> 90.3 tok/s.)

What the Q8_0 costs on 128 GB of RAM (125.7 GiB):

It does not load without the pins. 119.5 GiB of experts against ~120 GiB available with nothing else running.
With 11 + 11 GiB only in VRAM the engine takes ~97.5 GiB and ~19 GiB stay free (a prompt cache of 32 checkpoints and
10 GiB of parked conversations still fit; at least 6 GiB free once they are full).
One card is not enough. With only the 3090's pins ~108 GiB would have to fit in ~120: not runnable. And 11 + 11
is the ceiling: more pins leave the prompt path too few cache slots to borrow (with UD-Q4_K_XL at 11 + 11 an 18K
prompt already read at 532-1,078 tok/s instead of ~1,800).
No room for the PLE table in RAM (28.8 GB), so it keeps --ple-io direct (the Q4/IQ4 duals gain +33% on an 18K
Spanish prompt from ram).
Decode is bound by the RAM. Once both cards are full, ~1 GB of experts per token comes from the DDR4 (43 GB/s
measured): half UD-Q4_K_XL's speed whatever the prompt, and an accepted draft does not help (each extra token brings
its own experts). The two cards serve 87.8% of UD-Q4_K_XL's routed experts, 76.6% of UD-Q6_K_XL's and 68.1% of the
Q8_0's, and each miss is bigger: per token UD-Q6_K_XL reads ~2.9x and the Q8_0 ~4.4x what UD-Q4_K_XL reads from the
RAM. What read less from it won: --pcie-frac 0.25 (+5%), --spec-min-p 0.8 (+2.6%) and --feed-max 128 (the first
token of a short chat in 2.0 s instead of ~2.9).
The files cannot stay in the page cache next to a ~97 GiB arena, so every start reads the experts from the disk.
huihui's Q8_0 is a re-split GGUF: it needed the loader change of native dense: a re-split GGUF may name the architecture on every shard #15 (now in custom).
Route-resident routing (upstream 5df35dc, route-resident: one warp per token in the swap kernel Niko1221/Strata#1737) gives the Q8_0 +55% at margin 0.5, but costs as much quality as
going down to UD-Q4_K_XL (KL 0.024-0.038 against 0.006 between two plain runs), so we keep it off.

What we run instead: a UD-Q6_K_XL made from the Q8_0 (llama.cpp, unsloth's UD-Q6_K_XL recipe and imatrix; it keeps
the Q8_0's abliteration). It is as close to the Q8_0 as a second Q8_0 run is (KL 0.018 against 0.021) and 34% faster
(55.9 against 41.8 tok/s); its 101.7 GiB of experts fit in RAM, yet 6 + 6 GiB pinned still pays (54.2 -> 55.9) because
it reads less of the RAM. Details: README: a model whose experts do not fit in RAM.

eddoursul · 2026-10-12T18:57:23Z

Thank you. Merged on top of #19, with guards for what came after its base: refused with --disable-hit, --no-pool, --no-capture, --no-token-graph and the dumps (the token graph's hits are the pinned experts' only path), --verify-cache compares VRAM-only slots with the model's bytes, and the second GPU's reserve loop never gives up a pinned expert (too little VRAM for them now fails its start instead of a slot check later). With 4 GiB pinned on the 3090 at --adapt-every 0 the window hashes are identical to the run without, the arena 4 GiB smaller and STRATA_LOAD_VERIFY clean. Speed with the adaptive tiers, 4 GiB pinned on the 3090 (8 GiB the same): IQ3_S 0.8 / 2.3 / 2.9 / 2.2% less time a round (the 3090 tier copied 4-7x fewer MB over its x4 link), UD-Q4_K_XL 0.6 / 2.5 / 4.7% less but 13% more on the edit text (its hits fell from 80% to 61%: the frozen slots kept the profile's experts while the text wanted others). Opt-in; the RAM side (a Q5_K_XL arena past the pinning limit) is what it is for here.

(Measured on a Ryzen 7 9700X, 160 GB DDR5-5600, RTX 3090 + RTX 5070 Ti as the second expert tier, Windows 11; IQ3_S and UD-Q4_K_XL with this branch's configs, fixed text through --spec-follow, 2 rounds in both orders.)

…y in VRAM (--vram-pin-gib, --second-gpu-pin-gib) The copy of #13's commit under it resolved to this branch's (the Windows read loop's offset is `pos`). Guards for what came after its base: pins are refused with --disable-hit, --no-pool, --no-capture, --no-token-graph and the dumps (the token graph's hits are their only path), --verify-cache compares VRAM-only slots with the model's bytes, and the second GPU's reserve loop never gives up a pinned expert (too little VRAM for them fails its start). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

txiki739 and others added 6 commits October 12, 2026 13:30

txiki739 force-pushed the vram-pinned-experts branch from c71e3be to 9a9463a Compare October 12, 2026 12:21

eddoursul merged commit 2b419a5 into eddoursul:custom Oct 12, 2026

txiki739 deleted the vram-pinned-experts branch October 13, 2026 11:42

Navigation Menu

Repository navigation

Experts only in VRAM (--vram-pin-gib, --second-gpu-pin-gib) - #20

Experts only in VRAM (--vram-pin-gib, --second-gpu-pin-gib)#20
eddoursul merged 6 commits into
eddoursul:customfrom
txiki739:vram-pinned-experts

txiki739 commented Oct 12, 2026 •

edited

Loading

Uh oh!

txiki739 commented Oct 12, 2026

Uh oh!

eddoursul commented Oct 12, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

txiki739 commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Q8_0 and UD-Q6_K_XL on a 128 GB machine (what made these flags necessary)

Where this sits (measured today on your new custom)

This series (rebased onto e3727cb)

Tested on custom at e3727cb

Uh oh!

txiki739 commented Oct 12, 2026

Uh oh!

eddoursul commented Oct 12, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

txiki739 commented Oct 12, 2026 •

edited

Loading

Where this sits (measured today on your new `custom`)

This series (rebased onto `e3727cb`)

Tested on `custom` at `e3727cb`