Repository navigation
[Benchmark] record the host side in the standard report: measured PCIe per card, NUMA, memlock, GPU temperature and power #4610
Description
Area
Benchmark
What do you want to do?
Record in the standard report the host facts that change how the engine runs, next to the cards. Today hardware has the GPU names, VRAM and power limit, the CPU name, total RAM and the drive (hardware() in strata_cli/bench_facts.py). Three things I hit on my machine in the last days are not in it:
- The PCIe link. My two RTX 4060 Ti are PCIe 4.0 x8, and the engine measures them at 13.3 GB/s (
layer split auto: CUDA0 link 13.3 GB/s, probe best of four 11.6-13.3), about what a Gen3 x16 slot gives. The generation alone is misleading twice: by spec they are "Gen4", andnvidia-smiread at idle sayspcie.link.gen.gpucurrent1, because the link drops to Gen1 when idle. Only the measured figure says what the experts get ([Engine] Layer-major prefill on a CUDA layer split: 32K prompts 2.0x faster, 0.47x drive reads (Qwen3.8-Flash-Next, v0.2.4.2) #3976 is about this). ulimit -l. On v0.2.7.1 the PLE table goes to RAM andmlockfails here with a 15.7 GiB limit ([Engine] nv_defaults for sm_89 (Ada): the sm_86 switch set pays on an RTX 4000 SFF Ada 20 GB - scope agreement before a default change #3659). The line is in the engine.log of my standard run, but nothing in results.json says it, and the same machine with a higher limit would be a different run.- NUMA. Mine is one node, so it doesn't matter here, but [Benchmark] record the layer split (stages, split point, trim, residency) in the standard report #4495's machine has one card per socket.
Suggested, in an extensions.host block or next to hardware.gpus:
- per card: PCIe generation and width at max (nvidia-smi), and the link the engine measured, in GB/s;
- the NUMA node count, and the node each card hangs from when there is more than one;
RLIMIT_MEMLOCK, and whether an mlock failed in the engine log;- optional: GPU temperature and power, min/median/max over the run (the server already exports both, see [Other] Prometheus export and dashboard on a layer split: stage labels, per-card PCIe, hit counters #4496). This one needs sampling during the run, so it is more work than the others, and it can wait. It matters most on laptops and mini PCs, where the card slows down when it gets hot and the same machine gives different numbers cold and warm.
All of these are read, nothing on the machine is changed. The split topology and the expert residency per stage are #4495, so I left them out. RAM channels and speed would help too, but on Linux they need root (dmidecode), so I'd put them in missing rather than ask for sudo.
I'm not asking for a full hardware benchmark (STREAM, nvbandwidth and fio already exist), only for the facts that explain why two machines with the same cards give different numbers. A result without these variables can mislead whoever compares it, me included: on my machine the same cards would look like "Gen4" in one report and "Gen1" in another, and run at the speed of neither.
What do you do today instead?
I write these by hand in the README and attach the engine log.
Hardware: 2x RTX 4060 Ti 16 GB (PCIe 4.0 x8 each, layer split), Threadripper PRO 3975WX (1 NUMA node), 128 GB DDR4-2667, Pop!_OS 24.04. Version: v0.2.7.1 (46c152a1, engine 0.1.53), built here with CUDA 12.6 for sm_89. Model: Flash-Next UD-IQ4_XS.
I did this with an AI coding assistant: it read the bench code and my logs and drafted this text. The idea and the numbers I checked are mine.
Activity
udqe commentedon Oct 29, 2026
Support, from the other side of exactly this problem: we are the 2x RTX 4090 / TR3970X report the maintainers collected today (#4573, and #4345 from the laptop). Three concrete contributions to the list.
1. Our matrix is the worked example for "the generation alone is misleading twice". Same machine, same cards, only a BIOS setting changed: measured pinned copy 26.4 GB/s at Gen4 x16 vs 13-15 at Gen3 x16, and over that axis chunk-major prefill lost 16-22% while decode never moved (124.7-129.5 tok/s). Your 4060 Ti pair lands at 13.3 measured - the same rate class a Gen3 x16 slot gives our cards - from a completely different machine shape. Only the measured field can line those two stories up; "Gen4" would put yours next to our 26.4 and the reader would conclude the reports contradict each other.
2. One field your list is missing, from something we caught today: the effective layer_split K the engine picked at start.
layer_split: automoved our split 22 -> 23 -> 22 across three same-config starts in one day - silently, and it invalidated a single-variable A/B we had already analyzed before we noticed (layer split auto: CUDA1 runs layers 23-47vs22-47is one line in engine.log, invisible in results.json). Two lines in the same spirit as yours, both read-only, both start-up lines we would have paid to have recorded: the auto-picked K, and the arena pinning line (N slices pinned (X GiB)) - our whole +3.4% prefill reading fromSTRATA_ARENA_PIN_GIBonly makes sense because that line existed in the log.3. From the collected side: if results like ours land on a public benchmarks page, "same cards, different numbers" becomes the first issue readers hit, and your host block is what turns a confusing number into a legible one. We write by hand into READMEs today too (your "What do you do instead" is exactly this), and it is the first thing a reader skips.
One honest wrinkle for the "measured figure" side, and after counting today it is no longer a wrinkle: our startup probe reads 6.7 GB/s on CUDA0 in 10 of the 57 starts in our current engine log (~18%), while CUDA1 never once read low - CUDA0 is our monitor card (
Display Active: Enabled; CUDA1 is headless). The pattern is always exactly one step down (6.7 against 26.4 at Gen4, the same 6.7 against 13.2 when we ran the pair at Gen3 - two different expected values, the same misread). Your box is the clean control on a sibling platform - headless, Linux, and your 28 starts all read straight (your numbers from #3976). We tried to pin the trigger and could not: three controlled starts after 15-minute monitor-off idles all read 26.4/26.3/26.3, so whatever parks the link, ordinary 15-minute idle is not enough - the real 10 hits came from deeper cold states. Recording the measured number is what makes even this visible: a misread is a diagnosis when it is recorded, and an invisible one under a generation label.(AI-assisted as before: the model served by this repo's engine drafted this; the numbers and the drift anecdote are our logs.)
fioruccione commentedon Oct 29, 2026
Thanks @udqe, both fields go on the list. To say better why I opened this: I'm not after a more detailed leaderboard. I'd like a result to explain itself. Strata's speed comes from the whole box (CPU, RAM, link, NUMA, how much of the routed mass fits in VRAM, the split, which prefill path ran), not only from the cards. So two boxes with the same cards can give different numbers, and two boxes with very different cards can land closer than their specs suggest. Without these facts the second case can't be read either, and I'd like to use reports to decide which upgrade actually helps.
There is a practical side too. The installer already adapts the setup to the box, and that part works. But people read these threads and copy knobs. The same knob has opposite signs on different boxes:
--pipeline-windows 2is +7% for udqe and -14% in #4603, and the layer-major walk is +16% on their Gen4 pair and -26% on the V100s. A number taken alone, without the box it came from, pushes someone to change a setting that was already right for them.One measurement for the CPU side, from tonight on my box (2x RTX 4060 Ti 16 GB, Threadripper PRO 3975WX 32 cores, DDR4-2667, v0.2.7.1, UD-IQ4_XS, my production config). I run
--pcie-frac 0.00, so every expert missing from VRAM goes to the CPU pool, and in decode the cache hit rate is about 82-84%. I changed only--pool-workers, order 30 15 8 8 15 30, a fresh server per start:Putting #3976 and this thread together, this is what I'd call needed and what I'd call nice to have.
Needed (all start-up or log lines, read only):
--layer-split, see @agurrrrr in [Engine] Layer-major prefill on a CUDA layer split: 32K prompts 2.0x faster, 0.47x drive reads (Qwen3.8-Flash-Next, v0.2.4.2) #3976) an explicit "not measured" instead of an empty field;caches hold ... % of the routed massline, and the decode hit rate;--pcie-frac, since they decide how much of the miss work the CPU does;Nice to have: PCIe gen/width at max, temperature and power over the run, RAM channels and speed.
About the probe misreads: my two cards both report
Display Active: Enabled, and all 27 auto starts here read 13.3 GB/s on both. So on my box a display on the card is not enough to trigger it. That's one more reason to record the number at every start rather than trust a single reading.I have no comparable 4060 Ti vs 4090 measurement (same model, quant, context and phase), so I'm not putting a ratio on it. The point is that a report should carry what a fair comparison needs.
I did this with an AI coding assistant: it read both threads and my engine log, wrote the scripts and drafted this text. The idea is mine, and I ran the measurement and checked the numbers.
fioruccione commentedon Oct 30, 2026
Following up on the CPU table above with the rest of what I measured today on the same box (2x RTX 4060 Ti 16 GB, Threadripper PRO 3975WX, 8ch DDR4-2667, v0.2.7.1, UD-IQ4_XS, production config,
--pcie-frac 0.00). Every row is a fresh server per start, alternated order, decode at T=0.All of this depends on
--pcie-frac 0.00: every miss is computed by the CPU pool, so the CPU is in the decode path. With a higher share the misses go over PCIe and the link counts more than the cores. On this box the CPU path wins clearly: when I calibrated an older version the auto-probe picked 0.37 (40.8 tok/s) and 0.00 gave 50.3, with 32 cores, 8 channels and an x8 link per card. I'd expect it to pay off even more on a newer CPU with faster cores and DDR5, at least where the cards sit on a narrow link like the 4060 Ti's x8. With a wide Gen5 x16 link the balance could move the other way. I haven't measured either case, so it's worth checking on each box rather than copying my 0.00.This is one machine, and I'm not claiming the percentages travel. A modern 8-core with higher clocks, more IPC and DDR5 would not behave like 8 workers on one Zen 2 CCD. But one box is enough to show that the same cards, model and config can move by a fifth because of the CPU alone. That's why I think a report needs the CPU pool, the cores,
--pcie-fracand the hit rate next to the GPU name, otherwise two numbers can't be compared and people copy settings that only fit the box they came from.I did this with an AI coding assistant: it wrote the scripts, the small bandwidth program and drafted this text. The idea and the questions are mine, I ran the tests and checked the numbers.
fioruccione commentedon Oct 30, 2026
One more thing, after the measurements above: I checked what the standard runner (
strata bench) records today, at46c152a1, so it's clear which part of this is already covered.Already there:
/proc/cpuinfo(cpu_name, bench_facts.py#L130);ram_bytes, #L111);--pcie-frac,--pool-*,--layer-split,--vram-*land inoffloading(GROUPS, #L507).Not there, as far as I can see:
--layer-split autoit doesn't say which K was picked;caches hold ... % of the routed massline;dmidecode, so root, so probably an optional field the user fills in.With today's numbers that matters: on my box the pool worker count alone moved decode by up to 21%, and none of it would show in a report where the config uses the default workers. Two such reports would look like noise.
Most of the missing facts are start-up or log lines the engine already prints, and the format already has
extensionsfor data outside the contract, so I think they could go there without touching schema v1. I'm not opening a PR unless you think it's the right place; just pointing out the gap.I did this with an AI coding assistant: it read the runner code and drafted this text. I checked the references.
udqe commentedon Oct 30, 2026
fioruccione's CPU-pool table above is the sharpest argument in this thread, and our pair happens to sit at the opposite extreme of every axis he moved - so we ran the mirror experiment to see whether the same knob is even measurable on a rig at 97% cache hits. It is not, and we think that answers his own "should the percentages travel" question better than a yes.
Where the two boxes sit
--pcie-frac30 pool workers on logical processors 2,4,...60, the host thread on 0)CPU compute per token is ~1/7 of his. His 8-worker cliff (-12 to -21%) should be invisible here; the question was whether we could measure how invisible.
Protocol (alternating, pre-declared rule)
Fresh server per arm, A(w30)/B(
--pool-workers 8) alternating across sessions to cancel day-drift; decode = salted cold 150K prompt + 512 generated tokens per run, median of the engine's owngenerated ... (tok/s)lines; windows silent. Rule declared before looking: claim an effect only if B>A in >= 4 of 5 paired groups AND the pooled median gap exceeds +2%; else report "undetectable". (The engine's own docs for the flag say "Moves threads only, never a result" - this tests speed, not bits.)Actual (7 sessions, 44 decode runs, engine 0.1.54)
Pooled: A n=24 median 101.6 / B n=20 median 102.0 (+0.34%; means +0.37%). Sign split 2/4 - rule not met: workers 30 -> 8 is undetectable here (|effect| < ~1%).
The part worth keeping
Within-run spread was 87-115 tok/s (28%) against a between-arm gap of 0.3% - 44 runs could not separate the two arms, and our own same-config decode medians drifted 97 -> 111 across a day (documented in our #4573 addendum). Also related: at T=0 the answers themselves are not reproducible on this server (#4673), so a single decode median, let alone a single output diff, does not compare between boxes.
So the table this issue is asking for should carry what both boxes already print -
pool workers+ the logical list + host core, the hit-rate line,--pcie-fracand how it was set, the effectivelayer_splitK and the arena-pin line (both still missing from the standard runner, our earlier row) - plus each number's n and spread. Our 8-worker round produced a number (+0.34%); we would not have been entitled to report it as a finding, and the pre-declared rule is what stopped us. One box can move a fifth by the CPU alone (fioruccione), ours cannot feel the CPU at 97% hits: same knob, both answers real, only the recorded fields tell you which box you are reading.(AI-assisted as usual: the model served by this repo's engine ran all 44 runs and drafted this; every number is an engine-log line from the machine described.)
fioruccione commentedon Oct 30, 2026
Thanks @udqe, and thanks for taking the time to run 44 tests to check this from the other side.
Your results are exactly what I hoped this issue would bring out: the same setting can have a major impact on one machine and almost none on another, and both results are perfectly consistent once we look at cache residency and how much work actually reaches the CPU. About 1.7 missed experts per token on my box against 0.26 on yours is roughly 6-7 times less work for the CPU pool.
And it goes beyond decode. On my box layer-major prefill often can't start: in the production log it was turned down for "no VRAM for a super-chunk" (238-434 MiB free) on 104 of the 200 prompts longer than one 8192-token chunk. So on my box the prefill path itself changes from prompt to prompt. With 24 GB cards that is probably a different story. Every configuration has its own peculiarities, and a number without them doesn't tell you much.
The 82.9% vs 97.4% hit rate explains more than the GPU names alone ever could.
I think we've made a good case for recording the effective runtime configuration alongside benchmark numbers, rather than leaving readers to reconstruct it from logs.
And I particularly appreciate that you reported a 0.34% difference as undetectable instead of turning it into a performance claim. That's the kind of discipline that makes community benchmarks useful.
Thanks again for the work!
udqe commentedon Oct 30, 2026
Thank you - and your 104-of-200 line is the best argument this issue has heard: on your box the prefill path itself changes prompt to prompt depending on free VRAM, so even a prefill number carries your box in it before any of mine does.
One honest note about our own 44 runs so nobody over-reads the "undetectable" verdict: it rules out effects at the scale you measured - a -12 to -21% cliff would have been obvious on this rig even in single digits of runs - but it does not prove the effect is zero. This environment carries a +-10% day-drift at identical settings (the same addendum's own numbers: 123.6 -> 111.4 decode median, prefill flat), and within a window single runs ranged 87 to 115 tok/s (28% spread). Against a noise floor that wide, anything under roughly ten percent is unresolvable even at n=44: 44 trials were simply not enough trials to trust a small difference, which is a property of this box, not a doubt about yours. The same drift would have let us "discover" a fake +4% if we had stopped at two runs per arm and compared means - which is the whole reason the decision rule was written down before the data was looked at.
So the honest summary sentence for our row of your table is: "at 97% hit rate a pool this size is inside our +-10% floor at n=44; smaller effects cannot be seen here at all." If someone with a low-hit-rate box wants to run the same alternating protocol, the bar to clear on a drifting rig is about that: many more runs, or a rig quieter than ours.
(AI-assisted: the model served by this repo's engine ran all 44 runs, drafted this reply, and served as the subject of every number quoted.)
udqe commentedon Oct 30, 2026
Self-correction on our first comment in this thread (Oct 29, the one proposing the host-facts fields: #4610 (comment)): there we wrote "We tried to pin the trigger and could not... the real 10 hits came from deeper cold states." We can now pin it - and we also owe an apology about how we got there.
The apology first, specifically: the three controlled starts behind that sentence were a flawed experiment, and the sentence drawn from them was wrong in both directions. What we recorded as "starts after 15-minute monitor-off idle" was not that condition - the machine was entering sleep during the idle (one-minute screen timeout on the Ryzen power plan), and the failed first-start attempts after wake warmed the GPU before the three "clean" reads. So the 26.4/26.3/26.3 results tested nothing we designed, and the conclusion we published - "the real 10 hits came from deeper cold states" - is the opposite of what the next experiment showed: the misread concentrates in the FIRST start shortly after a SHORT sleep, while the deepest cold state we could produce read clean (0/1). We posted an artifact as a finding; sorry to fioruccione especially, whose clean 28-start control we cited in the same breath.
The corrected experiment, stated as protocol this time: engine stop -> 60 s -> 15 min sleep -> start within seconds of wake (W), alternating with engine stop -> immediate restart (N), with
layer_split=autoset temporarily so the probe line prints at all - it never appears in a fixed-split session, its own blindspot for anyone trying to reproduce us. The history it was designed against: 10 of 57 starts read CUDA0's link at 6.7 GB/s against a pinned 26.4, CUDA1 never, always the same fingerprint - CUDA0 is the display-attached card, and the number is always exactly half of the true generation's value (6.7 vs 26.4 on Gen4, 6.7 vs 13.2 during a Gen3 BIOS period).Actual: W 4/8 fired (50%), N 0/2, one longer-sleep session 0/1 (weak n, disclosed). Against our mixed historical rate of 18% the 4/8 is p~0.05 (binomial). Start #99 reads
layer split auto: CUDA0 link 6.7 GB/s -> miss cost x1.00besideCUDA1 link 26.4in the same boot at the same instant, identical config; the negative W starts read 26.4 on both cards. So the trigger on this box is: first start shortly after resume-from-sleep catches the display-attached card's link still ramping; the probe prints half the ramped width, misses cost nothing because the miss cost line is x1.00 in every firing, and a fixed-split config hides the whole symptom.Two consequences for the issue's ask:
probe value + start context + card display-attachedwould let readers tell these apart without our nine sessions of forensics.Caveats we keep attached: n=8 is small, the effect is display-card-only, the N arm is thin (0/2), and the pin that fixed the number in the first place (
--pcie-fracexplicit) remains our mitigation - we do not trust auto-probed link values on this box at all anymore.(AI-assisted: the model served by this repo's engine ran the reads, the protocol bookkeeping and this draft; every start number is from the machine's own engine log.)
fioruccione commentedon Oct 31, 2026
liaiwu commentedon Oct 31, 2026
+1 on the memlock line, with one thing your wording can be read as missing: the limit alone is not enough, you
also need whether an
mlockactually succeeded. On a start from a hand-written systemd unit the limit is8 MiB, the residency policy checks RAM only (
ple_resident_admit,include/strata/core/ple_residency.hpp:23) and admits the table anyway, and the kernel refuses. Two starts ofthe same unit and the same config are then two different runs, and both print a normal-looking start-up.
Measured on that pair: same machine (RTX 4090, 125.6 GiB), same config, arms alternating, medians of six runs
per side — decode 134.7 tok/s refused against 142.8 tok/s locked, +6.0%, Welch t=5.1, reproduced in two
independent arm pairs; on a second workload +2.6%, which six runs cannot separate. MTP acceptance unchanged.
Details in #4671.
The two start-up lines that say which state a run was in already exist, so nothing new has to be read off the
machine: the refusal at
src/kernels/ngram.cpp:461, and the pairedPLE table locked in RAM/PLE table loaded (not locked)atsrc/program/generate.cpp:4766.Mlockedin/proc/meminfoandVmLckin/proc/<pid>/statusgive the byte figure: 11 MiB against 26.83 GiB, which is the table.(AI-assisted as before: the assistant read the source and my logs and drafted this; the runs and the numbers
are mine.)
longstone78 commentedon Oct 31, 2026
Thanks for opening this issue. I ran a small exploratory
--pool-workerssweep on a third, fairly different machine: dual Haswell-EP, 2x RTX 5060 Ti 16 GB, a measured Gen3 x8 link per card, and a much lower decode cache-hit rate than the 4090 system discussed above.The short result is that moving from 11 to 18 workers improved mean decode throughput across two different prompts by about 8.4%, while moving from 18 to 22 added only about 1.1%. This is only
n=1per arm and the arms were not fully alternated, so I am reporting it as a machine-specific exploratory result, not a general performance claim.Host
6.8.0-138-generic, x86-64.numa_node=-1. This may be firmware/node interleaving, but I did not change the BIOS to investigate it.INTEL, product/boardX99, C610/X99 chipset. This is a generic identification rather than a useful model number.0000:02:00.0.0000:03:00.0.6.9 6.9 6.9 6.9for both cards.RLIMIT_MEMLOCK: 12,637,913,088 bytes (11.77 GiB; shellulimit -lreports 12,341,712 KiB).VmLck: 0 KiB. This old build does not print the newerPLE table locked in RAM/loaded (not locked)line. It does printexpert arena: cudaHostRegister PORTABLE ok.Strata, model and effective runtime configuration
Niko1221/Strata.477ac13cbe8e582c20484871ce459d20d82df09d, described asv0.1.45-dirty.BUILD.json: version0.1.45, sourceb95137167bd9b751, CUDA architecture[120], vision backendcpu, vision source13d9deaf827f14ef.setup.py;data/draft_vocab_it.binis also present as an untracked local data file. The engine itself was not patched for this worker test.--spec 4 --spec-min-p 0.5.20; CUDA0 runs layers 0-19 and CUDA1 runs layers 20-47.--vram-reserve-mib 9216 --vram-reserve-later-mib 6656. These were chosen so occasional vision workloads can coexist on the cards.pcie_frac 0.19versus the engine default 0.55. I did not pin--pcie-fracexplicitly.The only variable intentionally changed between arms was
--pool-workers.Protocol and limitations
/v1/chat/completions,temperature: 0,max_tokens: 512, non-streaming.finish_reason: length.Exact prompts
Systems prompt:
Code prompt:
Results
Systems-design prompt, 142 prompt tokens + 512 generated tokens
The 18-worker and 22-worker arms followed the same generated path according to the engine counters. On that matched path, 18 -> 22 moved decode from 54.7 to 55.5 tok/s, about +1.5%. Prefill did not improve with worker count.
Python-code prompt, 131 prompt tokens + 512 generated tokens
Again, the 18-worker and 22-worker arms had identical draft/cache/routing counters. On that matched path, 18 -> 22 moved decode from 50.7 to 51.1 tok/s, about +0.8%.
Compact summary
These are means across two different prompts, not repeated trials of one workload:
The average HTTP wall time was 13.800 s at 11 workers, 12.934 s at 18, and 12.980 s at 22. The decode improvement above 18 workers was mostly offset by ordinary prefill/start-of-request variation in this very small sample.
One discarded warm-state observation
Before correcting the protocol, I ran the code prompt at 22 workers as the second request on an already-used 22-worker server. That request reported 55.6 tok/s, 76.3% cache hit and 243/329 accepted drafts. I excluded it from the table because it was not comparable with the fresh-server 11- and 18-worker arms. After a complete restart, the proper first-request 22-worker result was 51.1 tok/s with the exact same draft/cache/routing counters as the fresh 18-worker result.
That discarded point is still a useful reminder that adaptive/warm state can be larger than the 18 -> 22 worker effect on this machine.
My interpretation
This box is CPU-sensitive because its decode cache-hit rate is only about 68.5-74.3%, and the link probe selects a low
pcie_fracof 0.19, so a meaningful share of miss work remains on the CPU pool. Reducing the pool to 11 workers costs about 6-10% decode on these two prompts. However, this old dual-socket Haswell system is already close to a plateau by 18 workers: using 22 of its 24 physical cores adds only about 1% mean decode in this sample.I left the production configuration at 22 workers because the other workloads on this machine are sporadic, but 18 workers appears to be the better efficiency/coexistence point.
The result supports the main point of this issue: GPU names alone would not explain this run. The measured link, cache residency/hit rate, auto-selected
pcie_frac, CPU model, effective worker placement, fixed split and warm/adaptive state are all needed to interpret the number.If this machine is useful as the requested third configuration, I can run a stricter pre-declared alternating protocol with more fresh starts per arm. I would probably compare 11, 18 and 22 workers again, record every startup fact, and report medians/spread rather than means.
fioruccione commentedon Oct 31, 2026
Thanks @longstone78, this is exactly the third machine the thread needed, and the host facts are all there.
In return, here is everything I run on my box (2x RTX 4060 Ti 16 GB, Ada sm_89, Gen4 x8 probing 13.3 GB/s, TR PRO 3975WX, 0.2.7.1 built for sm_89), with what each piece measured here. Your cards are Blackwell and your link is half of mine, so treat every number as mine, not as a prediction for yours.
Two things I'd look at first on your box
--vram-reserve-mib 9216 --vram-reserve-later-mib 6656you keep about 15.6 GB of 32 free for vision, and you end up with 2,772 resident experts. With--vram-reserve-mib 700and the defaults for the rest, my two cards hold 7,615 resident experts (thetoken graph hit pathline) and the decode hit rate sits around 83%, against your 68-74%. Your IQ3_S experts are smaller than my UD-IQ4_XS ones, so the same VRAM should hold even more of them on your box. I run--visiontoo, with the encoder on the CPU ("gpu": falsefor vision in the config). If vision is occasional, this is probably your biggest lever, bigger than the cores.--pcie-frac. When I calibrated, the auto-probe picked 0.37 (40.8 tok/s) and0.00gave 50.3, +23%. With a 6.9 GB/s link the CPU path should look even better on paper, but your pool is already close to a plateau at 18 workers, so here it really has to be measured.Engine options in production (besides pack, model, profile and vision):
Environment, all A/B-tested here one change at a time:
The sm_86 set was tuned for Ampere. On Ada it paid, and on Blackwell nobody has measured it as far as I know, so I'd test it as one arm, not copy it.
Things that did not pay here, to save you the time:
--trim-stage-weights: +25% experts and +6 points of hit rate, but decode flat and prefill +1-4%;--pipeline-windows: +8-11% at T=0 with a lot of variance, -7% on a 20K summary, +18% energy;One home experiment, not in production numbers: VRAM memory offset +2000 (nvidia-settings) gave about +12% decode on a bench and +13% on real work at 120K+ context, with 0 Xid in 5.5 hours. My cards are GDDR6, yours are GDDR7, and I can't read memory temperature on either, so that one is at your own risk.
Not tried here, but measured by others:
STRATA_ROUTE_PRIOR(0.2.7+). In #4345 and #4573 it moved the hit rate from about 87% to 97-98% and decode by 30-50%, at a measured quality cost (#4345 has the KL table). With your hit rate it might be the strongest of all, but it trades quality for speed, so I'd measure both.On method, the only thing that worked for me: one change per arm, a fresh server per start, the first run discarded, arms alternated, and three or four runs per arm at least.
I did this with an AI coding assistant: it read my config and notes and drafted this text. The settings and the numbers are from my machine, and I checked them.
udqe commentedon Oct 31, 2026
Answering the four-item list from our side - the 4090 box already quoted in this thread (the n=44 pool row) is ours, and everything below was already in our logs before the list existed, so no new runs were needed. Windows, so the power-plan variable fioruccione suspected applies to us.
1. pool workers: default 30 vs
--pool-workers 8, alternating sessions, pre-declared rule. 44 decode runs, 5 paired fresh-server sessions, salted cold 150K prompts: pooled median w30 101.6 vs w8 102.0 (+0.34%), sign split 2 of 4, rule not met - undetectable at our hit rate (aggregate 97.4%). Within-run spread on this box is 87-115 tok/s (28%), so sub-1% is unresolvable here at any n we ran. Per-window hit rates in those sessions: 96.6-99.3%. This is the deliberate opposite pole from fioruccione's rig (82.9% hits, 1.7 missed experts per token vs our 0.26, -12..-21% at 8 workers): same knob, opposite sensitivity, and the only reason we can say "no effect" honestly is that his box says "big effect" with the protocol written down first.2. layer-major on our config: it declines, with the exact line, 199 times. Every long-prompt start in the last two days prints
layer-major off for this prompt: no VRAM for a super-chunk (1171 MiB free)-style (free MiB varies, 1100-1800), 199 occurrences, never a walk start. Config: 2x 4090 24 GB, fixed split K22,--kv-resident 32768, expert cache ~5,070 slots, margin96/972. So on this box "does the walk start on a long prompt" = "no, and the refusal line plus its free-MiB figure is the whole answer"; we can't contribute a walk measurement at 24 GB/cards without gutting the expert cache.3. link probe across start contexts: 119 link lines, 15 misfires, trigger pinned. All from the
layer_split=autoera (our production split is fixed 22, where the line never prints - its blindspot, noted in our self-correction). The misread is discrete and reproducible: first start within ~a minute of resume-from-sleep catches the display-attached CUDA0 at exactly half its true generation value (6.7 vs 26.4 on Gen4), 4 of 8 such starts, 0 of 2 plain restarts, 0 of 1 deepest-cold boot; CUDA1 never; miss cost always x1.00.4. host facts, filled from the list: Windows 11 (26200), driver 610.88, both cards RTX 4090 24 GB (24,564 MiB), Gen4 x16 both, probe 26.4 GB/s steady, display on CUDA0; TR 3970X 32c/64t, 8x16 GB DDR4-3200 tightened timings (204.8 GB/s theoretical, memcpy never run); pool workers 30 (default on this CPU),
--pcie-frac 0.20set explicitly; engine 0.1.54 (release 0.2.8), will note separately what changes if we move to 0.1.55 - we are holding production at 0.1.54 exactly because of the arm-delivery lesson this thread keeps teaching; power plan AMD Ryzen Balanced, sleep enabled (our 26.4-era "idle" mislabels came from exactly this - the apology comment),RLIMIT_MEMLOCK-equivalent: Windows page-lock, ours succeeds (PLE table locked in RAMat every boot; liaiwu's refused-state is Linux-unit specific as far as we can see).One asymmetry worth stating since the list asks for comparability: our box contributes the HIGH-hit-rate corner (97%+) and the quiet-link corner (x16 Gen4), fioruccione's the low-hit corner, longstone78's the low-link low-hit corner. If a fifth machine shows up it should ideally be low-hit AND multi-NUMA - the one axis nobody covers is >1 NUMA node with experts, since both our box (1 node) and his (single socket) hide it.
(AI-assisted as usual: the model served by this repo's engine read its own logs and drafted this; all numbers are from runs this box made, several of them in the thread above.)
Burntgogi commentedon Nov 1, 2026
A timing-field example from a two-RTX-3090 Qwen comparison
One additional case for this reporting proposal: a reanalysis of our saved 0.2.4.2 versus 0.2.9 comparison gives different changes for native-time and client-SSE decode rates. The client figures were already included in #4147; the native-time means below are newly calculated from the same captured requests, not new model runs. This is a custom deployment comparison, not a standard benchmark submission or a confirmed regression report.
Setup: Linux x86_64, Ryzen 7 5700X, 64 GB-class RAM, two RTX 3090 cards with 24 GiB each; Qwen3.8-Flash-Next IQ3_S in a layer split. Configured context 147,456. Baseline: source 0.2.4.2 / Qwen engine 0.1.48. Candidate: source 0.2.9 / Qwen engine 0.1.55, both official CUDA13 packages. The engine reported 7 pool workers and
pcie_frac=0.19in all four starts.The same weights, initial expert-profile seed, device order and base options were used. Process order was old → new → new → old, two fresh starts per version. Each version had six short-input and six long-input speed samples: 188/512 input/output tokens and 131,134/1,024 respectively. All speed outputs reached the requested length cap. Each group includes initial and prefix-reused requests; it is not a cold-only workload. The two metrics below use the same samples, not independent sets.
Important non-default condition: both arms used the same unchanged experimental-speed control vector, strength 1.0, layers 4–44,
project/per-layer, with the related sampling toggle enabled. No vector-off arm was measured. These are not stock-profile results.Both are arithmetic means of per-request rates. Native-time rate is
predicted_n / (predicted_ms / 1000)from the captured native timings. Client rate is(completion_tokens - 1) / (SSE completion wall time - time to first nonempty reasoning or answer delta). They cover different boundaries. The percentage gaps must not be subtracted and called server overhead; these records do not isolate the cause.Actual path selection also matters. Both candidate starts announced layer-major prefill but declined it for the initial long prompts, for example:
That is configuration context, not a new fallback bug or an explanation of the decode difference. Related fallback behavior is already discussed in #3976.
Would it help for the report to retain both named timing definitions and their token counts, the first/last delta and SSE-end timestamps, and the requested versus effective prefill path with its fallback reason? This would keep native and client rates from looking interchangeable and help select the next profiling step before attributing a version effect.
Limits: small sample, no statistical significance claim; source server and native engine changed together. Clocks, temperature and OS page-cache state were not fully controlled. No additional profiling or vector-off test was run for this reanalysis. Length-capped coding outputs are not a coding-correctness test. No DeepSeek sizing result or historical cache-admission failure is being resubmitted here.
Only generic hardware/model facts, necessary technical settings, aggregate measurements and one numeric log excerpt are included. No credentials, local paths, device identifiers, raw request/response content, projection files or full logs are attached.
fioruccione commentedon Nov 1, 2026
Three more A/B runs from my box, same method as before: fresh server per start, arms alternated, one warm-up request discarded, then prose x2 and code x2 (400 tokens, T=0) and a 20K-token summary (300 tokens). Following @Burntgogi's point, the rate is the engine's own timing (
predicted_per_secondin the response timings), not a client SSE rate. Box: 2x RTX 4060 Ti 16 GB, TR PRO 3975WX, 128 GB, 0.2.7.1 built for sm_89, production config, VRAM memory offset +2000 in every arm.1. The memlock line (@liaiwu)
Thanks for it, my box was in the refused state at every start. The default memlock limit here is 15.7 GiB (16,860,569,600 bytes, nothing set in limits.conf or systemd) and the PLE table is 26.8 GiB, so all 54 starts in my log printed
PLE table mlock failedandloaded (not locked), withVmLck0.Six starts, U L L U U L, the only change being the soft memlock limit of the server set with
prlimit. Every L start printedPLE table locked in RAMwithVmLck28.1 GB, every U start the refusal.+0.2%, most of it from the last start: no measurable difference here. My guess, not measured, is that with 128 GB and no memory pressure the table stays resident anyway. So I agree the line is what tells you the state, but the effect is per machine: +6% on your 4090, +0.2% here.
I also ran the
--ple-io directarm @alexanderi96 suggested in #4671: six more starts, D R R D D R, R being production as it starts (ram chosen by the residency policy, not locked), D the same config plus--ple-io direct(probe 58-98K rows/s, "fine"). NVMe SSD.Short prompts are flat (-0.8%), but at 20K direct is slower in all three starts: -3.2% decode, -4.4% prefill, and the ranges don't overlap. And on this box it gives no RAM back: RSS drops by the table, but MemAvailable is the same, because the unlocked table is file-backed pages the kernel counts as reclaimable anyway. So here ram unlocked is the best of the three: same speed as locked, faster than direct at long context, and no memory pinned.
2. Same box, same config, a different model file
@longstone78 runs ISTA-DASLab's GSQ-RCO IQ3_S, so I downloaded the same file (sha256 checked against setup.py) and ran it against my production UD-IQ4_XS. Same config, only pack,
--nativeand tokenizer changed. Six starts, Q P P Q Q P.The first Q start is discarded: the IQ3_S was on a USB disk, the engine reported ready while its pages were still being read from it (57 GB resident, threads in D state), went 13-36 tok/s, hit the 60 s stall watchdog and was restarted. The other two Q starts were normal, so that's 2 Q against 3 P.
Smaller experts, more of them in VRAM, less work on the CPU, and the code prompt gains the most. Draft acceptance was the same for both files. On the 20K summary layer-major declined in both arms (
no VRAM for a super-chunk, 367 MiB free with IQ3_S, 355 with UD-IQ4_XS), the same refusal @udqe and @Burntgogi report. What this does not measure is quality: IQ3_S has fewer bits per weight, and I'd expect more mistakes from it, so +17.6% is only worth something if the answers stay good. I haven't tested that yet.For the thread, the same file shows the point again (@Burntgogi runs it too, on two 3090s): with @longstone78's reservations it holds 2,772 resident experts on two 5060 Ti, here 10,012 on two 4060 Ti with the same 16 GB each. The engine version, link and protocol differ too, so I wouldn't compare the tok/s, only the residency. Model and hardware have to be chosen together, and the same model file can give very different numbers depending on the box and the config.
I did this with an AI coding assistant: it found the refusal in my logs, wrote the A/B scripts and drafted this text. The runs are from my machine, and I checked the numbers.