**DeepSeek V4 Flash UD-IQ2_XXS on Windows — RTX 5090 D v2 24 GB + 128 GB RAM: first Windows report; 29-36 tok/s decode at 32K, exact recall verified with filled contexts up to 1,008,128 tokens (256K/512K/1M all 3/3 needles)**

First Windows report for DeepSeek V4 Flash, since the docs said it "has been run on Linux only" and no Windows numbers are in the issues yet. One context note first: your repository returned 404 for about two days (Oct 13-14) and came back today (Oct 15) — that is the only reason this report is late; the machine was ready on the 14th. Might be worth checking what happened with the hosting.

It runs on Windows with the ready-made engine, no local build, and it is fast enough for daily use. 24 GB is below the 32 GB you tested on, so fewer experts stay resident — the numbers below still clear everyday-use speed.

Hardware

GPU: NVIDIA GeForce RTX 5090 D v2, 24 GB, single card
CPU: Intel Core i7-14700KF (28 threads, AVX2)
RAM: 128 GB DDR5
Storage: model on a local NVMe SSD
Driver: 616.92

Software

OS: Windows 10, 64-bit
PolyStrata repo at commit c566ff4 (2026-10-14)
Engine: ready-made release binary, strata-poly v0.1.44.1 (BUILD.json: src bd887f5d6b808672, CUDA 13.0, sm_120 included) — no local build
CUDA runtime: NVIDIA pip wheels (cu13)

Model

unsloth/DeepSeek-V4-Flash-0731-GGUF, UD-IQ2_XXS, three files, 90.9 GB, sha-checked by setup (--gguf-dir, files already on disk)
dspark-DeepSeek-V4-Flash-0731-BF16.gguf (11 GB) is present in the folder but unused — see the draft-model question below
Text only

Settings

Install: python setup.py --family deepseek --model UD-IQ2_XXS --gguf-dir <folder> --port 8090 --context 32768 --no-browser --yes (setup's "has been run on Linux only" risk prompt is answered by --yes)
Default run: --max-context 32768 --conversation-cache-mib 8192 --threads 13
Context tiers tested by editing the run config to --max-context 262144 / 524288 / 1048576 (all inside the trained 1M, no rope scaling)
--vram-reserve-mib 1024 added for the second benchmark run (see below)

Results

polystrata.py benchmark (greedy, 256-token replies, thinking at its default "medium" effort, warm engine). Machine totals at peak, not the engine's alone.

First run, default (no VRAM reserve), expert profile from its first learning pass:

prompt	tokens in	TTFT s	prefill tok/s	decode tok/s	MTP right
short	14	2.50	5.6	36.0	0.711
long-2000	2547	8.46	303.4	28.9	0.404
long-8000	10067	11.67	688.3	34.1	0.650

Second run after adding --vram-reserve-mib 1024 (1 GiB kept free for the desktop; profile warmed):

prompt	tokens in	TTFT s	prefill tok/s	decode tok/s
short	14	0.59	23.8	35.0
long-2000	2547	5.43	469.3	30.4
long-8000	10067	10.02	805.4	31.8

The 1 GiB reserve costs nothing measurable here and keeps the desktop usable alongside the engine — worth a mention in the docs for small cards.

Long-context recall (3 needles planted at 10% / 50% / 90% depth, greedy, thinking off via `reasoning_effort: none`)

max-context	prompt tokens	read time	avg read tok/s	needles found
262144	251,866	640 s	393	3/3
524288	503,920	2,040 s	247	3/3
1048576	1,008,128	7,294 s	138	3/3

(Your 512K run on the 32 GB card: 487,529 tokens in 1,923 s — the 24 GB card reads the same scale at roughly the same pace. At 1M the quadratic indexer cost is fully visible: the second half of the prompt reads far slower than the first.)

The dspark draft model (question)

The repo ships dspark-DeepSeek-V4-Flash-0731-BF16.gguf (11 GB) but the engine does not run it yet; guessing comes from repeated conversation text only. Is draft-model support for DSV4 on the roadmap? Reasoning it is worth it here: the routed experts are the slow part and they are read from RAM, so verifying several draft tokens per step amortizes the same expert reads across positions — MoE on a memory budget is exactly the case where speculative decoding pays. The current text-repeat guesser shows 0.40-0.71 "MTP right"; a trained drafter should do better.

Thanks, that's the first time DeepSeek V4 Flash ran on Windows, and on a 24 GB card at that. I'll add your numbers to docs/COMMUNITY_BENCHMARKS.md with the next docs update. The 1M run with all three needles found is a good result, and the 1 GiB reserve note is useful for small cards.

The 404: the repository was unreachable on the 13th and 14th because of GitHub, it's back now.

The dspark draft model: not yet. For DeepSeek V4 the engine only guesses from text the conversation repeats, which is the 0.40 to 0.71 you saw. A trained drafter should do better for the reason you give, and it's on the list, but I have no date for it.

OK, I will continue to pay attention to the follow-up updates, test and submit them.

docs: DeepSeek V4 Flash runs on Windows, by a user's report

Closing: your numbers are in docs/COMMUNITY_BENCHMARKS.md and docs/DEEPSEEK.md as of 0.1.44.1-poly.4. The dspark drafter is listed under what's missing in DEEPSEEK.md. New results are welcome in a new issue.

Data point: a model file larger than RAM — UD-IQ4_XS at 137 GB on a 128 GB machine — collapses decode to 1-2 tok/s, about 10x below what llama.cpp held paging a similar model from the same SSD. JSON attached.

Setup as in my earlier reports: RTX 5090 D v2 24 GB, 128 GB DDR5, NVMe SSD, Windows 10, threads 13, engine 0.1.44.1-poly.5 (no local build). Model: DeepSeek V4 Flash 0731 UD-IQ4_XS, 4 splits, 136.66 GB total (sizes byte-identical to the ModelScope listing). Config: --max-context 262144, --conversation-cache-mib 8192, --vram-reserve-mib 1024; expert-profile reused from my IQ2_XXS runs without errors. Load: 110 s, 895/11008 routed experts in VRAM (10.8 GiB), down experts converted MXFP4→Q5_0 (1.38 GiB of RAM per layer). dspark not supported yet, so this is the plain run.

The numbers (bench, 256 output tokens)

tier	prompt tok	decode tok/s	prefill tok/s	TTFT s	RAM peak GiB	GPU util %
short	14	1.0	0.9	15.3	127.61	3.1
long-2000	2547	1.4	22.8	111.8	127.56	8.3
long-8000	10039	2.1	53.4	149.7	127.61	19.8

RAM peak sits at 127.6 GiB — the physical ceiling — and GPU utilization stays at 3-20% the whole way: the card is starved, waiting on experts that have to come back from the SSD. MTP acceptance actually rises with depth (0.25 → 0.67 → 0.91), so the drafter is fine; it is pure I/O starvation.

Two references from the same machine

UD-IQ2_XXS (90.9 GB, fits with ~20 GB to spare), same engine, same day: decode 31.6 / 31.3 / 32.7 tok/s, RAM peak 108 GiB.
llama.cpp (older test of mine): UD-Q4_K_XL at 155 GB, paged from the same NVMe, held ~14 tok/s decode.

So under the same oversubscription, llama.cpp's plain mmap streaming held roughly 14 tok/s while the expert-shuttling path lands at 1-2 — about an order of magnitude apart. Not a complaint: a file bigger than RAM is clearly a hard regime, and the engine offers no knob for it today (no mmap / lock / RAM-residency cap in --help). But if a RAM-residency cap or a streaming read mode for the experts ever makes the list, this is exactly the case it would fix.

The "down experts converted MXFP4→Q5_0 (1.38 GiB of RAM per layer)" line in your load log was the engine making its own copy of those layers' experts at load, because nothing on the card could multiply MXFP4. That copy is the engine's own memory, so Windows cannot drop it and read it back from the file the way it does with the file's pages, and on a machine where the file is already bigger than RAM it left that much less room for the file's pages. poly.6 multiplies MXFP4 on the card and on the CPU directly, so the conversion and the copies are gone. On my RTX 5090 test machine with UD-IQ2_XXS the engine's own RAM went from 5,959 MiB to 341, the 24K prompt read went from 1,000 to 1,091 tokens/s, and writing speed stayed the same.

I haven't run a file larger than RAM, so I can't say how much of the gap to llama.cpp this closes. The rest of it, experts read back from the SSD, is still there. If you run UD-IQ4_XS again on poly.6, the same benchmark JSON would show it. One more thing: on RTX 50 cards the prompt's rows for those layers are now quantized to FP4 the way llama.cpp does it, so greedy text can differ a little from poly.5.

The IQ3_XXS numbers are welcome when you have them.

Follow-up, as promised: UD-IQ3_XXS (104.2 GB) fits and runs well on the same 128 GB machine — full 512K needle at 231 tok/s avg (3/3), just 4.6% under my IQ2_XXS numbers, with the same quadratic decay shape. So on this box: 104 GB is the sweet spot, 137 GB is unusable. JSONs attached.

Same machine (RTX 5090 D v2 24 GB, 128 GB DDR5, NVMe SSD, Windows 10, threads 13, 0.1.44.1-poly.5). Model: DeepSeek V4 Flash 0731 UD-IQ3_XXS, 4 splits, 104.21 GB total (byte-identical to the ModelScope listing). Config: --max-context 524288 (deliberately capped at 512K — no headroom for 1M on 128 GB RAM), --conversation-cache-mib 4096, expert-profile reused from the IQ2_XXS pack without errors.

The numbers

test	IQ3_XXS 104.2 GB	IQ2_XXS 90.9 GB (same day)
32K bench, warm: decode (short / long-2000 / long-8000)	27.4 / 29.5 / 30.7 tok/s	31.6 / 31.3 / 32.7
100K needle (3 needles at 10/50/90%)	501.8 tok/s, 3/3	600.3 tok/s, 3/3
512K needle	519,757 tok in 2,245.7 s = 231.4 tok/s, 3/3	242.5 tok/s, 3/3

So the whole-quality step up from 2-bit to 3-bit costs 5–16% depending on depth — on the depth that matters for long reads it's only 4.6%.

The 512K decay map (sampled every 10 s via the metrics API, ~224 samples)

	IQ3_XXS (this run)	IQ2_XXS (my earlier run)
opening rate	~705 tok/s @ 8K	~713 tok/s @ 10K
crosses 400	@ 93K	@ 137K
crosses 300	@ 193K	@ 198K
crosses 200	@ 280K	@ 301K
crosses 150	@ 378K	(n/a)
floor	95–157 tok/s @ 506–518K	~114 tok/s @ 464K

Same shape as before: a fast start, then the quadratic scoring tail takes over. The IQ3 curve tracks the IQ2 one closely, just shifted slightly earlier — consistent with a bigger model paying a bit more scoring cost, not with any memory pressure (see below).

Occupancy during the 512K read

System RAM peaked at 119.8 GB of 127.8 GB (≥8 GB free the entire run; never any thrashing)
VRAM free stayed flat at 920 MiB the whole way; expert_slots flat at 1,367 — same story as before: the tail is indexer scoring cost, not memory pressure

For completeness on the oversubscription side (my previous post): with the 137 GB UD-IQ4_XS the same sampler saw RAM pinned at the 127.6 GiB ceiling and decode at 1-2 tok/s — so the practical ceiling on 128 GB is exactly this 104 GB tier.

Attaching: needle-dsv4-512k-iq3xxs.json, needle-dsv4-100k-iq3xxs.json, bench-dsv4-iq3xxs-32k-warm.json.

bench-dsv4-iq3xxs-32k.json
bench-dsv4-iq3xxs-32k-warm.json
needle-dsv4-512k-iq3xxs.json

Follow-up on the UD-IQ4_XS report from yesterday, retested on 0.1.44.1-poly.6 as you suggested.

Same machine (RTX 5090D V2 24G, 128 GB RAM, NVMe SSD with a 192 GB page file on it), same model files (UD-IQ4_XS, 4 shards, 136.66 GB, not re-downloaded), same config as yesterday's run (--max-context 262144, --conversation-cache-mib 8192, --vram-reserve-mib 1024, --threads 13, expert-profile reused from dsv4-ud-iq2_xxs).

The fix works: the load log has zero "down experts converted from MXFP4 to Q5_0" lines now (poly.5 printed one per layer, 43 layers, ~58 GiB of fixed RAM copies in total). Load time dropped from 110.2 s to 28.3 s, and 992 of 11008 routed experts fit in VRAM (10.8 GiB).

Bench, 256 output tokens, same protocol as yesterday (poly.5 → poly.6):

prompt	tokens in	decode tok/s	prefill tok/s	TTFT s	RAM peak GiB	GPU %
short	14	1.0 → 3.4	0.9 → 1.1	15.3 → 13.3	127.55	5.1
long-2000	2547	1.4 → 2.2	22.8 → 49.6	111.8 → 51.4	127.74	11.0
long-8000	10018	2.1 → 2.8	53.4 → 99.5	149.7 → 80.3	127.57	21.1

JSON attached (bench-dsv4-iq4xs-poly6-32k.json).

So removing the conversion copies helped a lot — decode is 1.6-3.4x faster and prefill roughly doubles. But the basic mismatch is still there: the file (136.66 GB) is bigger than physical RAM (127.8 GB), RAM still peaks at ~127.6 GiB with the mapped pages, the oversold part still random-pages from SSD, and decode stays at 1-3.4 tok/s — still about 4-5x below plain llama.cpp mmap streaming the same-class quant (our old UD-Q4_K_XL 155G run, ~14 tok/s on this machine).

IQ3_XXS (104.2 GB) remains the practical ceiling here: needle 512K at 231.4 tok/s, data in my previous comment.

One thing I noticed: with the copies gone the engine is much leaner, so if you ever consider a "resident-RAM cap / streaming expert read" mode for the >RAM case, this quant might become viable on 128 GB machines — happy to test if it lands.

Stopping note: after the bench I let the machine settle until paging activity was zero, then stopped the engine; RAM released cleanly (114 GB free), no reclaim storm.

bench-dsv4-iq4xs-poly6-32k.json

Thanks for both runs, and for retesting IQ4_XS on poly.6 so quickly. The IQ3_XXS and IQ4_XS numbers are now in docs/COMMUNITY_BENCHMARKS.md and docs/DEEPSEEK.md.

A file bigger than RAM is listed in DEEPSEEK.md under what's missing. I haven't measured where the time goes in that case, so no promise yet. Two things in the engine probably hurt there. After load, a background thread reads one byte of every page of the experts, which helps when the file fits and pushes hot experts out when it doesn't. And the conversation cache in your config can take up to 8 GiB of the same RAM.

One question that would help: which model was the llama.cpp UD-Q4_K_XL run at 155 GB? If it wasn't DeepSeek V4 Flash, a token reads different experts there and the 14 tok/s isn't directly comparable.

The exact llama.cpp version was so long ago that I can't recall it, and I didn't record it at the time — fair point on the comparison. The card was this same RTX 5090D V2 24G (128 GB RAM, same NVMe). I can re-download the UD-Q4_K_XL and rerun a clean test with the full setup recorded, and send you the detailed data afterwards.

That would be great, thanks. The most useful version is both engines on the same DeepSeek V4 Flash file: llama.cpp and poly.6 on UD-IQ4_XS (or UD-Q4_K_XL if you'd rather download that one), same prompts, with llama.cpp's version and full command line recorded. For the poly.6 run, --conversation-cache-mib 0 would take the conversation cache out of the RAM picture. Then any gap is the engine's, and I'll know where to look.

Head-to-head done as you suggested: same files, same prompt, both engines, full command lines recorded.

Setup: RTX 5090D V2 24G, Ryzen/Intel 14700KF (13 threads), 128 GB RAM, 4 TB NVMe, Windows 10. llama.cpp b11706 (0.7.0-dev, commit f74e2fa6a, official prebuilt CUDA 13.4). PolyStrata 0.1.44.1-poly.6 with --conversation-cache-mib 0 as you specified. Fixed Chinese prompt (51 tokens), temperature 0, 256 output tokens, identical across all runs.

Commands:

# llama.cpp (experts on CPU/mmap, attention + dense + KV on GPU)
llama-server.exe -m <shard1> -ngl 99 --n-cpu-moe 999 --threads 13 --port 8090
# with the drafter
llama-cli.exe -m <shard1> -md dspark-DeepSeek-V4-Flash-0731-BF16.gguf -ngl 99 --n-cpu-moe 999 --threads 13 ...
# poly.6 engine args
--model <shard1> --max-context 262144 --conversation-cache-mib 0 --vram-reserve-mib 1024 --threads 13

Decode results (256 output tokens):

Model	llama.cpp bare (cold → warm)	llama.cpp + dspark draft	poly.6 (conv cache 0)
UD-Q4_K_XL, 155 GB	7.7 → 10.6 / 10.4 tok/s (server timing: 9.45 tok/s, 105.77 ms/token)	2.0 / 2.4	1.4 → 2.2
UD-IQ4_XS, 137 GB	4.2 → 8.6 / 8.6	4.4	(bench this morning: 3.4 / 2.2 / 2.8 at 32K/2000/8000-prompt)
UD-IQ3_XXS, 104.2 GB (added)	5.3 → 11.0 / 10.9	5.2	poly.5 needle 512K: 231.4 (different protocol, reference only)

Occupancy: llama.cpp held 22.5-23.6 GiB VRAM (no spill), process working set grew 14 → 85 GB as the mmap page cache filled, GPU util 10-15%. poly.6 held ~23 GiB VRAM (861 of 11008 routed experts in VRAM, 10.7 GiB), working set 103 → 110 GB, GPU util 0-10%. No MXFP4→Q5_0 conversion lines anymore on poly.6 — the native path is working.

Two side notes:

llama.cpp warns "tensor overrides to CPU are used with mmap enabled - consider --load-mode none" — not viable here, the 155 GB file doesn't fit in 128 GB RAM.
The dspark draft hurts in this regime on our machine: 2.0-2.4 vs 9.45 bare (Q4_K_XL), 4.4 vs 8.6 (IQ4_XS), 5.2 vs 11.0 (IQ3_XXS). Draft verify rounds seem to amplify the expert-fetch cost rather than save it. (Mystery from my earlier "~14 tok/s" memory solved while writing this: that was UD-IQ3_S at 119 GB run on llama.cpp b11118 back in late September, 13.1-16.4 tok/s as seen through desktop clients, since deleted — a different quant, a different llama.cpp build, and client-side numbers, so it doesn't compare against this table.)

Follow-up, same day — the old 14 tok/s mystery fully solved, and it changes the dspark story:

My September setup for IQ3_S (from the router config I still had) was not -md at all. It was:

--load-mode none (full RAM load — IQ3_S at 119 GB just fits in 128 GB, so experts are RAM-resident, zero paging)
the router preset key spec-type = draft-dspark (llama.cpp's native DSpark speculative path, not the generic --model-draft)
override-tensor exps=CPU, KV q4_0, -c 262144

I reproduced that exact recipe today on b11118 with IQ3_XXS (104.2 GB, the biggest DSV4 quant that still fits in RAM):

recipe (b11118, full RAM load)	decode, 256 tokens
bare (no spec lines)	13.5 / 13.5 tok/s
with `spec-type = draft-dspark`	10.5 / 9.6 tok/s

So: bare 13.5 matches the old notes exactly ("prose 12.8-13.2"), and DSpark on prose is actually slightly negative — its gain was only ever on fixed-format tasks (counting-style: 18-21 tok/s in the old notes). Two structural points fall out of this:

The fast recipe requires the model to fit in RAM (--load-mode none). The 155 GB / 137 GB 4-bit quants can't, so on a 128 GB machine DSpark can't help them no matter the usage — the >RAM regime is paged either way.
In the >RAM regime, every draft-verify round re-reads the paged experts, which is why -md made things worse everywhere I tried it.

So the honest summary for the >RAM comparison stands: llama.cpp 8.6-9.5 vs poly.6 1.4-2.2 (4-5x), and no draft mechanism closes it on this hardware.

So with everything held constant, llama.cpp holds 8.6-9.5 tok/s streaming experts through mmap in the >RAM regime, while poly.6 lands at 1.4-2.2 — a 4-5x gap even after the MXFP4-native fix removed the conversion copies. Happy to test anything else that might close it.

Thanks, that's the comparison I needed, and it settles where the 14 tok/s came from. So with the file bigger than RAM, on the same files, prompt and machine, llama.cpp holds 8.6 to 10.6 tok/s and poly.6 1.4 to 2.2. That's a real gap in strata-poly. I'll look at what the engine does differently when it pages, starting with the pass that reads every page of the experts after load and the way the CPU experts read pages that aren't in RAM, and post here when there's something to try. I'll also correct DEEPSEEK.md and COMMUNITY_BENCHMARKS.md, which still compare against the 14.

The dspark numbers help too: in this regime every verify round reads the paged experts again, and on prose it didn't help even with the whole file in RAM.

Merge PR simplexai-labs#21: Q4_0 KV with Hadamard rotation (--kv q4_0…

Record PR simplexai-labs#21's rebased head as merged (its change is a…

Cross-engine data point: DeepSeek-V4.1-Flash-1017 XQ on the same machine (RTX 5090 D v2, 128 GB) — DSpark acceptance goes net-negative on the 1017 XQ build too, matching the draft-hurts observation in this thread. Posted for cross-reference; easy to move or drop if it doesn't belong here.

First, transparency on provenance, since this is not PolyStrata's engine:

The 1017 checkpoint is quantized in XQ format (per-expert quant types llama.cpp can't load), so it only runs on the Xolotl engine — https://github.com/Indras-Mirror/Xolotl (Indras-Mirror's fork, the 1017 line). Model repo: https://huggingface.co/IndrasMirror/DeepSeek-V4.1-Flash-1017-XQ-Uncensored. The numbers below come from that engine, built from source today (CUDA 13.1 + -arch=sm_120a; note for anyone on RTX 50: CUDA 12.8's ptxas rejects the MXFP4 templates' mma with block scale, and vswhere -latest can pick VS 18 whose MSVC nvcc refuses — pin VCVARS to VS 2022).
I am also filing the same findings with the Xolotl tracker, since the questions (draft acceptance on 1017) are theirs to answer. This post is only the cross-engine comparison, because it lines up exactly with what was observed in this thread: on this machine, the draft hurts when expert residency can't absorb the verify cost.

Machine (same as my previous reports here): RTX 5090 D v2 24 GB, i7-14700KF, 128 GB DDR5-4800, Windows 10, model + Engram on NVMe. Model: DeepSeek-V4.1-Flash-1017-XQ-Uncensored V2, 152.2 GB joined and sha256-verified; Engram = official 1017 shards 47/48 + 48/48 read in place (https://www.modelscope.cn/models/deepseek-ai/DeepSeek-V4.1-Flash-1017). Engine load 49.7 s; arena 72 GiB; 14.38 GiB of VRAM expert slots (~1,500 experts, same design point as the docs).

Prefill is not the problem (fresh prompts, streaming TTFT):

prompt	tokens in	prefill tok/s
long-8000	4,941	230
long-32000	19,619	485
long-128000	78,321	515
long-200000	122,350	509

Decode with and without the DSpark draft (128-token gens, streaming):

config	8K decode	32K decode	tokens/pass (log)
DSpark depth 4 / conf 0.6 (engine defaults)	8.3	8.1	1.77–2.42
DSpark depth 2 / conf 0.75	7.3	6.9	1.37–1.63
no DSpark	12.8	13.0	—

Why my decode is ~13 tok/s while this thread's V4 Flash 0731 runs 29–36 — three stacked differences, not an engine problem, and this thread already contains the proof for the biggest one:

RAM fit is the dominant factor. The 29–36 tok/s rows here are UD-IQ2_XXS at 90.9 GB — the whole model fits inside 128 GB RAM with ~20 GB to spare, so routed experts are RAM-resident and decode is pure compute. My 1017 XQ build is 152.2 GB > 128 GB RAM: 72 GiB arena + PCIe tiering, so every pass streams non-resident experts over PCIe. This thread measured exactly that cliff on PolyStrata too — UD-IQ4_XS at 137 GB (bigger than RAM) collapsed to 1–2 tok/s from pure I/O starvation. Same physics, both engines.
It is also a different generation, not just a bigger file. The 29–36 rows are V4 Flash 0731 (the July V4 line); mine is V4.1-Flash-1017 (October), which per the Xolotl docs decodes slower than its own predecessor across the board (12.3–18.6 tok/s on a 3090 for 1017 vs 13–22 for the V4.1 September line). My no-draft numbers sit right in the 1017 band.
Acceptance never sustains break-even on 1017 (~2.4 tokens/pass needed): 1.37–1.86 on Chinese short chat (greedy and temp=1 alike), 2.35–2.9 on repetitive English filler, degrading as the arena rebalances — so the draft costs more than it saves at any depth tested.

That gives two engines independently showing the draft flipping negative when residency can't absorb the verify cost — same GPU, same RAM class. Whatever the fix is (higher acceptance, cheaper verify, or better residency), it has to clear that bar.

One question for you, since this is your thread: the official 1017 is deepseek-ai/DeepSeek-V4.1-Flash-1017 (MXFP4 + Engram, same two-shard Engram pattern as 0731). Is V4.1-1017 on PolyStrata's roadmap? If it ever lands, the same needle tests on the same machine would give a clean Xolotl-vs-PolyStrata comparison on the checkpoint that actually matters going forward.

Next on the same machine: the direct predecessor of 1017 — DeepSeek-V4.1-Flash (September GSQ line) — is downloading now and will be tested on Xolotl with the same protocol (prefill sweep + decode with/without DSpark, depth 4 / conf 0.6, the engine's tuned defaults for that checkpoint). Two caveats up front: it is 152 GB on a 128 GB machine (streams like the 1017 build does), and it is the checkpoint the Xolotl docs make their boldest claims on (13–22 tok/s on a 3090, 23–30 on a 4090). So this run answers what this thread can't: do those 20–30 decode numbers reproduce on a 128 GB / 5090 D v2 box at this RAM ratio? Your #21 covers the V4 0731 line rather than V4.1, so there is no direct PolyStrata number to compare against — but if the 20–30 class holds on Xolotl, the V4.1 family would be a natural next target for PolyStrata too. Follow-up to be posted here either way.

If this post is unwelcome here for any reason — off-topic, unwanted, or stepping on anything — say the word and I'll delete it, no questions asked.

Thanks, this belongs here fine, and it lines up with the larger-than-RAM case in this thread. V4.1-Flash-1017 isn't on the list yet. It was asked for in the vote thread (#16) a few hours ago, so I've counted your question there too. I haven't looked at what changed from the 0731 checkpoint, so I can't say yet whether the current DeepSeek path loads it or needs engine work. Your September-line follow-up is welcome here either way.

September (V4.1-Flash GSQ) follow-up on RTX 5090 D v2, Windows — full numbers as promised

Hi VecSzn, following up on my earlier cross-engine post about the 1017 checkpoint. You mentioned you hadn't revisited 0731 and welcomed September feedback — here it is. Same machine throughout, so the three checkpoints are directly comparable:

Run	Checkpoint	Engine	DSpark	decode tok/s
1017 GSQ (XQ)	V4.1-Flash-1017	Xolotl 0.1.44 Windows sm_120a	off	12.8 (8K) / 13.0 (32K)
1017 GSQ (XQ)	V4.1-Flash-1017	same	depth4/0.6	8.3 (8K) / 8.1 (32K)
September GSQ	V4.1-Flash-September	same	depth4/0.6 (your #21 setup)	8.4-14.0 (32K-256K ladder)
September GSQ	V4.1-Flash-September	same	off (control)	9.6-10.6

September details (32K ctx, your #21 methodology)

Segment	prompt tok	prefill tok/s	TTFT s	decode tok/s
short	70	7.9	8.8	8.4 (cold first request)
long-2000	1277	76.1	16.8	13.3
long-8000	4943	226.0	22.5	10.2

256K ladder (prefill stays ~500 tok/s all the way up)

prompt tok	prefill tok/s	decode tok/s
19619	479.5	9.2
39185	497.0	9.3
78321	507.7	9.5
117459	508.8	14.0
156593	500.4	9.3

Retrieval integrity (3 needles at 10%/50%/90%)

9/9 exact recall through 163K-token prompts, positions correct. On this engine the expert-streaming design errs toward "slower but right", and that held at every depth.

Acceptance / occupancy

MTP/DSpark acceptance: 2.27 tokens/pass overall (per-request 1.72-3.13). On 1017 the same config gave 1.37-1.86 — September drafts clearly better, but decode is still ~9-14, not 20+. A 72→88 GiB RAM-arena A/B (file-tier experts −21.6%) changed decode by ~0, so the limiter is draft acceptance economics, not expert I/O.
DSpark on/off control (September): on 8.4/13.3/10.2/9.2/9.5, off 8.5/9.6/10.3/10.6/10.5 — roughly a wash (on 1017 DSpark was a clear -35% negative). Unexpected: September's bare decode is slower than 1017's bare decode.
Occupancy at load/bench: 21.9/24.4 GiB VRAM, 99.3 GB process private RAM (of 128 GB), model 163 GB streamed from NVMe.

Takeaway for your RAM table

163 GB model + 24 GB VRAM + 128 GB RAM = Windows stream-from-disk regime; it works (9/9 needles, stable 500 tok/s prefill to 256K) and lands at the low end of your 3090 September line (13-22) on decode, with roughly 2x your prefill. Happy to run any specific prompt set you want for the table.

Thanks for the full numbers. To keep the thread clear for others: these are Xolotl's runs of V4.1-Flash, not PolyStrata's. PolyStrata doesn't run the V4.1 checkpoints yet, and the "3090 September line (13-22)" is from Xolotl's docs, not from this repository. The part that carries over is what you already found: with a file larger than the RAM, decode on that machine sits around 9-14 tok/s on both engines, and the draft only pays off when its acceptance clears the cost of checking it. V4.1 stays counted in the vote thread (#16).

PolyStrata 0.1.45-poly.2 on the same RTX 5090 D v2 24 GB box — DeepSeek V4 Flash UD-IQ2_XXS and UD-IQ3_XXS, engine update alone vs engine + `--skip-weight 0.095` + `--stage-read-mib 4096`, at 128K/256K/512K

2026-10-20. Follow-up to my posts above (UD-IQ2_XXS first Windows report, UD-IQ3_XXS 512K, the over-RAM IQ4_XS data point). Same machine: RTX 5090 D v2 24 GB, i7-14700KF, 128 GB DDR5, model on NVMe, Windows 10, threads 13.

UD-IQ4_XS follow-up will come later — the 137 GB files from my earlier test have since been deleted; I'll re-download and test it when disk time allows.

Setup

Engine: PolyStrata 0.1.45-poly.2, prebuilt strata-poly-windows-x64-4fa9bedab2538cec (BUILD.json: 0.1.45, src 36c7ead, CUDA 13.0, archs incl. 120). Repo serve/server.py, streaming, numbers from the server's usage fields. No draft model (dspark still unused).
Switches: --skip-weight 0.095 --stage-read-mib 4096 — both are GLM-side features per the 0.1.45-poly.2 notes, so this is also a "do they help the deepseek4 path at all?" test.
Configs: --max-context 524288 --conversation-cache-mib 4096 --vram-reserve-mib 1024 --threads 13; expert-profile reused from the IQ2_XXS pack (as before).
Method: each tier is the first request of that length in the process (cold read); decode = completion_tokens / (total - ttft), reasoning_effort: none; single runs. The r2 rows in my raw JSON are prefix-cache hits and are excluded. Note: my 512K tier reads ~517K tokens (I capped the filler under the 524,288 context), so it is not directly comparable to my old 512K needle protocol (519,757 tokens, 3 planted needles) — the old numbers are quoted as reference only.

UD-IQ2_XXS (90.9 GB) — engine 0.1.45-poly.2

tier	tok in	bare prefill tok/s (ttft s)	bare decode	switches prefill tok/s (ttft s)	switches decode
128K	133,467	498.1 (268.0)	28.7 (46 out)	535.6 (249.2)	30.4 (46)
256K	266,930	493.8 (540.6)	42.2 (256)	511.5 (521.8)	38.4 (256)
512K	517,173	448.1 (1154.1)	35.5 (46)	308.6 (1675.9)	36.1 (46)

Occupancy (bare): VRAM ~23.2-23.5G, engine RAM ~92-98G throughout. Load ~25s.

UD-IQ3_XXS (104.2 GB) — engine 0.1.45-poly.2

tier	tok in	bare prefill tok/s (ttft s)	bare decode	switches prefill tok/s (ttft s)	switches decode
128K	133,467	351.7 (379.5)	39.6 (256)	460.0 (290.1)	37.1 (256)
256K	266,930	459.3 (581.1)	43.2 (256)	479.0 (557.3)	41.8 (256)
512K	517,173	296.5 (1744.3)	29.9 (46)	294.6 (1755.8)	32.7 (256)

Occupancy (bare): VRAM ~23.0-23.2G, engine RAM ~104.7-104.8G. Load ~25s.

Against my old poly.5 numbers (same file, same box)

metric	poly.5 (Oct 16, needle protocol)	poly.2 bare	poly.2 switches
IQ2_XXS 512K avg read	242.5 tok/s	448.1 (+85%)	308.6 (+27%)
IQ3_XXS 512K avg read	231.4 tok/s	296.5 (+28%)	294.6 (+27%)
IQ2_XXS 32K warm decode	31.6-32.7	—	—
IQ3_XXS 32K warm decode	27.4-30.7	39.6 (cold, 32K)	37.1 (128K)

The plain 0.1.45-poly.2 update is a large long-read win on the deepseek4 path: +85% on IQ2_XXS and +28% on IQ3_XXS at 512K scale, before any switches.

What the switches do on the deepseek4 path

Short/medium tiers: small win. IQ3_XXS 128K prefill 351.7 → 460.0 (+31%), IQ2_XXS 128K 498.1 → 535.6 (+7%); 256K roughly +4%.
512K tier: net negative on IQ2_XXS. 448.1 → 308.6 (-31%) — consistent with --stage-read-mib borrowing VRAM from the expert cache, which hurts most exactly where most experts stream from RAM/SSD. IQ3_XXS 512K is flat (296.5 → 294.6).
Decode is a wash everywhere (within noise). The deepseek4 bottleneck stays expert streaming, not routing, so --skip-weight has nothing to bite on the way GLM does.
Practical takeaway on 24 GB + 128 GB: for 512K reads run the deepseek4 path bare; the switches only pay below ~256K, and modestly.

Notes

My 512K tier (517K tokens, no needles) is a throughput probe, not the recall protocol — if you want the 3-needle 512K rerun on poly.2 for apples-to-apples with my Oct 16 post, I can run it.
Decode rows with 46-token completions are noisy (GLM/DSV4 short eos); the 256-token rows are the trustworthy ones.
UD-IQ4_XS: will follow up later (re-download first).

Thanks, this is useful, especially the 512K row.

Most of the gain over your poly.5 numbers likely came with poly.6, not poly.2: that release computes DeepSeek's MXFP4 experts on the card as they are, instead of keeping converted copies (the load dropped from 110 s to 28 s on your machine then). poly.2 didn't change the deepseek4 path itself.
The switches aren't GLM-only, they're engine-wide; the release notes named GLM because that's where they were measured. Your result fits how --stage-read-mib works: during a read it borrows VRAM from the expert cache so a layer's experts go to the card in one round. At 128K that's a win; at 512K the attention cache already takes most of the card, so what it borrows is expert cache the read needed, and IQ2_XXS loses 31%. So for very long DeepSeek reads on 24 GB, leaving it off is the right call, as you concluded. --skip-weight only changes which RAM experts get computed during decode, so a wash there fits too.
No need for the needle rerun on my account; the throughput probe answers what changed. The IQ4_XS follow-up whenever you get to it.

Thanks — that explains it. poly.6's MXFP4-on-GPU was the part I was missing, so the big 512K jump was already there before poly.2. Noted that the switches are engine-level, not GLM-specific — and thanks for confirming the --stage-read-mib mechanics at 512K; running deepseek4 bare at 512K on the 24 GB card from now on. IQ4_XS re-download is on my list, I'll post the numbers when I have them.

Re: IQ4_XS retest on 24 GB + 128 GB RAM — oversubscription confirmed, and one engine-level observation

Hi, as promised I re-downloaded UD-IQ4_XS (4 shards, 136.66 GB, byte-verified against ModelScope) and re-ran it on my 5090D v2 24 GB / i7-14700KF / 128 GB DDR5 / Windows 10 box with 0.1.45-poly.2. Short version: the result confirms my Oct 16/17 findings, and comparing three engines side by side on the same machine points at one specific gap I'd like to flag.

1. poly.2 retest (same config family as my IQ2/IQ3 runs: --threads 13, --vram-reserve-mib 1024, expert-profile reused from dsv4-ud-iq2_xxs)

The moment the model loads, RAM pins at ~127.6 GiB and decode sits at single digits tok/s — same as poly.5 (1.0–2.1) and poly.6 (2.2–3.4) before. I didn't run the full 3-level bench this time since the dashboard numbers were unambiguous and identical to the earlier runs; happy to run the formal JSON pass if you want it.

2. The new data point: two engines, same machine, same >RAM regime

Engine	Model	Size (oversubscription)	decode tok/s
PolyStrata poly.6	DSV4-0731 UD-IQ4_XS	136.7 GB (~9 GB over)	2.2–3.4
PolyStrata poly.2	DSV4-0731 UD-IQ4_XS	136.7 GB (~9 GB over)	single digits
Xolotl (ds41_generate)	DSV4.1-1017 V3 GGUF	163 GB (~35 GB over)	12.1–13.0

The engine with the largest oversubscription is 4–5x faster than PolyStrata on the identical hardware. That gap is too big to be explained by quant format or model version alone — Xolotl's advantage holds across 8K/32K/128K contexts (12.8 / 13.0 / 12.1 tok/s, DSpark off).

3. The new piece of evidence (referencing my Oct 16/17 reports)

The mechanism itself is what I reported on Oct 16/17: no mmap/mlock/RAM-residency-limit parameter (--help checked in full), the OS paging the files on demand once the expert cache exceeds free RAM, random-access page-ins feeding the expert-to-VRAM pipeline while the GPU sits at 3–20% utilization — and my Oct 16 suggestion of a "RAM residency cap + sequential streaming" mode stands.

What is new today is the controlled comparison: Xolotl on the same machine, same >RAM regime, with a much larger shortfall, holds 4–5x the throughput by evidently streaming from the SSD in a controlled way instead of trusting the OS page fault path. That is the missing data point I didn't have when I made the Oct 16 suggestion — it shows the streaming approach isn't theoretical, another DeepSeek engine already does it at 12-13 tok/s at 163 GB.

So I'd like to renew that Oct 16 suggestion with this evidence attached: a "RAM residency cap + sequential streaming" mode for the expert cache (cap what the cache pins at, stream the rest from the SSD in large ordered reads instead of relying on page faults) would make every quant larger than RAM usable on 128 GB machines — IQ4_XS 137 GB would land somewhere around the Xolotl numbers instead of 2–3 tok/s.

Question: is that something you'd consider, or is the expert-cache VRAM-pinning architecture fundamentally tied to "everything either fits in VRAM+RAM or pages randomly"?

4. Housekeeping

IQ2/IQ3 numbers from yesterday stand; on 128 GB machines IQ3_XXS remains the practical ceiling, as you said.
My poly.2 retest config: --max-context 524288; context size made no difference to the oversubscription slowdown, the bottleneck is purely the weights exceeding RAM.
Can delete/re-open as you see fit — the actionable part is section 3.

Thanks for re-downloading and re-running it. The side-by-side makes the gap concrete: 12-13 tok/s from Xolotl at about 35 GB over the RAM, against 2-3 here at about 9 GB over, on one machine.

It's the gap docs/DEEPSEEK.md lists under "What is missing" since your first report, and your reading of the cause is right: strata-poly reads the experts through the OS file cache, so once the file is larger than the RAM every miss is a page fault, and the reads are small and in random order.

It isn't tied to the architecture. The expert cache already moves experts to the card in its own order; what's missing is reading the ones outside the RAM from the file in large ordered reads into a buffer of a capped size, instead of leaving it to page faults. That is a real piece of engine work, though (its own reader and buffer, and a choice of what stays resident), and I haven't started it, so I can't give you a date. When I do, the numbers in this thread are the target. No need for the formal JSON pass.

Totally understand — no rush at all. Engine work of that scale takes the time it takes, and knowing it's already on the docs/DEEPSEEK.md radar is good enough for me.

I'll keep testing on this 128 GB machine as new versions and quants land, and the moment there's anything relevant to this thread — engine updates, oversubscription numbers, whatever is useful — I'll post it here first thing.

Thanks for the quick and clear answers as always.

I am planning to put qwen4exp on strata-poly later, and its not just the check, we will need new kernels for QSA and the n-gram PLE table, but I don't know a version or date yet, also there is no flag to get past that check because its in the code. I am also planning to work on SSD optimization for strata-poly, and once we port it qwen3.8 will get it too, until then qwen3.8 will run on Strata's engine in PolyStrata like how you run it now.

Thanks for the quick reply — no rush at all, the SSD work sounds like exactly the right priority. Whenever qwen4exp lands we'll gladly test it on our box (128 GB RAM + RTX 5090D V2 24 GB).

One data point for the roadmap: we just measured the MXFP4+MTP build (Qwen3.8-Flash-Next-MXFP4-ngramQ8-NextN.gguf, 118 GiB — MXFP4 routed experts, attention/router/shared expert at Q8_0/F32, MTP head built into the file) on Xolotl: 118.6 tok/s decode median on short prompts (peak 128.9; 122.7 on a 2k-token prompt), vs 66.6 tok/s for the same model's Q4_K_XL on the same rig. Footprint at that speed: ~22 GiB of the 24 GiB VRAM, the remaining experts sitting in system RAM with plenty of headroom. So the payoff for strata-poly should be worth the QSA/PLE kernel work — expert streaming over this exact file with a bounded resident budget would be our use case.

**DeepSeek V4 Flash UD-IQ2_XXS on Windows — RTX 5090 D v2 24 GB + 128 GB RAM: first Windows report; 29-36 tok/s decode at 32K, exact recall verified with filled contexts up to 1,008,128 tokens (256K/512K/1M all 3/3 needles)** #21

Description

Hardware

Software

Model

Settings

Results

Long-context recall (3 needles planted at 10% / 50% / 90% depth, greedy, thinking off via reasoning_effort: none)

The dspark draft model (question)

Activity

VecSzn commented on Oct 15, 2026

biurquezjoselina-design commented on Oct 15, 2026

VecSzn commented on Oct 15, 2026

biurquezjoselina-design commented on Oct 16, 2026

The numbers (bench, 256 output tokens)

Two references from the same machine

Next

VecSzn commented on Oct 16, 2026

biurquezjoselina-design commented on Oct 17, 2026

The numbers

The 512K decay map (sampled every 10 s via the metrics API, ~224 samples)

Occupancy during the 512K read

biurquezjoselina-design commented on Oct 17, 2026

VecSzn commented on Oct 17, 2026

biurquezjoselina-design commented on Oct 17, 2026

VecSzn commented on Oct 17, 2026

biurquezjoselina-design commented on Oct 17, 2026

VecSzn commented on Oct 17, 2026

biurquezjoselina-design commented on Oct 19, 2026

VecSzn commented on Oct 19, 2026

biurquezjoselina-design commented on Oct 20, 2026

September (V4.1-Flash GSQ) follow-up on RTX 5090 D v2, Windows — full numbers as promised

September details (32K ctx, your #21 methodology)

256K ladder (prefill stays ~500 tok/s all the way up)

Retrieval integrity (3 needles at 10%/50%/90%)

Acceptance / occupancy

Takeaway for your RAM table

VecSzn commented on Oct 20, 2026

biurquezjoselina-design commented on Oct 20, 2026

PolyStrata 0.1.45-poly.2 on the same RTX 5090 D v2 24 GB box — DeepSeek V4 Flash UD-IQ2_XXS and UD-IQ3_XXS, engine update alone vs engine + --skip-weight 0.095 + --stage-read-mib 4096, at 128K/256K/512K

Setup

UD-IQ2_XXS (90.9 GB) — engine 0.1.45-poly.2

UD-IQ3_XXS (104.2 GB) — engine 0.1.45-poly.2

Against my old poly.5 numbers (same file, same box)

What the switches do on the deepseek4 path

Notes

VecSzn commented on Oct 20, 2026

biurquezjoselina-design commented on Oct 20, 2026

biurquezjoselina-design commented on Oct 20, 2026

Re: IQ4_XS retest on 24 GB + 128 GB RAM — oversubscription confirmed, and one engine-level observation

1. poly.2 retest (same config family as my IQ2/IQ3 runs: --threads 13, --vram-reserve-mib 1024, expert-profile reused from dsv4-ud-iq2_xxs)

2. The new data point: two engines, same machine, same >RAM regime

3. The new piece of evidence (referencing my Oct 16/17 reports)

4. Housekeeping

VecSzn commented on Oct 20, 2026

biurquezjoselina-design commented on Oct 20, 2026

VecSzn commented on Oct 24, 2026

biurquezjoselina-design commented on Oct 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

DeepSeek V4 Flash UD-IQ2_XXS on Windows — RTX 5090 D v2 24 GB + 128 GB RAM: first Windows report; 29-36 tok/s decode at 32K, exact recall verified with filled contexts up to 1,008,128 tokens (256K/512K/1M all 3/3 needles) #21

Long-context recall (3 needles planted at 10% / 50% / 90% depth, greedy, thinking off via `reasoning_effort: none`)

PolyStrata 0.1.45-poly.2 on the same RTX 5090 D v2 24 GB box — DeepSeek V4 Flash UD-IQ2_XXS and UD-IQ3_XXS, engine update alone vs engine + `--skip-weight 0.095` + `--stage-read-mib 4096`, at 128K/256K/512K