Skip to content

Multi-GPU performance is ~2x lower than Francesco Albano fork despite the changes being merged into v0.1.40 #1352

New issue
New issue
Closed as not planned
Closed as not planned
Multi-GPU performance is ~2x lower than Francesco Albano fork despite the changes being merged into v0.1.40#1352

Description

@chemical12
chemical12
opened on Oct 7, 2026

Hi,

I noticed in the v0.1.40 release notes that several of the multi-GPU improvements from the Hardin22/Strata-DualGPU fork were incorporated into Strata.
However, I am seeing a significant performance difference when running the same model and the same two GPUs.

Hardware:
[0] RTX 4070 Ti SUPER 16 GB
[2] RTX 5060 Ti 16 GB

128 GB system RAM
Linux Debian

Model:
Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S

Strata configuration
I am using:

--expert-cache auto
--prefill auto:32768
--spec 4
--spec-min-p 0.5
--mtp ...
--max-context 262144
--kv int8
--kv-resident 32768
--vision
--vram-reserve-mib 700
--remote-expert-opt
--pipeline-windows 2

GPU: [0, 2]
layer_split: auto

With Niko1221/Strata v0.1.40, I get approximately: ~40 tok/s decode

With the Hardin22/Strata-DualGPU fork, using the same model and the same two GPUs, I get approximately: ~80 tok/s decode (110 tok/s when coding)

I also tried explicitly enabling:
--resident-experts
--adapt-async 1
--pipeline-windows 2
in v0.1.40, but this did not reproduce the Hardin22 performance.

There is also a significant difference in prompt processing speed depending on the GPU order.
With: GPU: [0, 2] a prompt of approximately 6k tokens takes around 50 seconds to process.
If I change the order to: GPU: [2, 0] the same ~6k-token prompt takes only around 15 seconds.

However, the decode speed is essentially identical in both configurations (around 30-40 tok/s).

Activity

  1. ztsvvstz1337 commented on Oct 7, 2026

    @ztsvvstz1337
    ztsvvstz1337
    on Oct 7, 2026

    Hi I might have some insight on the prompt processing.
    Its heavily PCIe lane speed dependent for example on my machine:

    x16 -> Transfers around 26gb/s -> 2000-3000 tps
    x4 -> ~ 5gb or less ~1000tps
    x2 -> super slow 1-2gb/s -> only 200-500 TPS

  2. Noctorngaming commented on Oct 7, 2026

    @Noctorngaming
    Noctorngaming
    on Oct 7, 2026

    Hi I might have some insight on the prompt processing. Its heavily PCIe lane speed dependent for example on my machine:

    x16 -> Transfers around 26gb/s -> 2000-3000 tps x4 -> ~ 5gb or less ~1000tps x2 -> super slow 1-2gb/s -> only 200-500 TPS

    I have a 4070 on 4.0 x16, and v100 on 3.0 x4 and I get around 100-150 prefill

  3. Niko1221 commented on Oct 7, 2026

    @Niko1221
    Niko1221
    on Oct 7, 2026
    Owner

    Thanks for the numbers, and @ztsvvstz1337 for the PCIe notes. Here is what is in v0.1.40.x from that fork and what is not, from the commit log.

    In (with Francesco Albano as the author): the resident RAM mode on a layer split (fe9c10c), --vram-reserve-later-mib (5c4105c), the asynchronous adaptive expert tier --adapt-async 1 (b437b8e), the MTP drafter chain as one launch (8ba583f), pipelined verify windows --pipeline-windows (bfa532e), pipeline-windows beside the async tier (13eee45, #905, in 0.1.40.2), the CPU-side kernels (AVX2 Q8_K activation quantizer, AVX-VNNI forms, the gathered i-quant decode), and each card keeping only its own layers' weights (--trim-stage-weights, which also works under auto).

    Not the same as in the fork: in the fork several of these are on by default for two GPUs; here they are opt-in (--adapt-async 1, --pipeline-windows 2). A merge of the code is not a merge of the defaults, so the same model on the same cards does not run the same path unless the flags are given. Also not taken over: the automatic card order (the fork puts the faster card last in the split when layer_split is auto; the engine here keeps your order, which would explain the big gap between [0, 2] and [2, 0] on your prompts: the 4070 Ti SUPER should probably be the last stage), the fork's hybrid-CPU worker default (here setup only recommends it), the display-aware default VRAM reserves, and the "fewer host waits" commits for the window (we have not found a matching commit).

    One thing in your config that matters: --pipeline-windows is switched off by --remote-expert-opt (the helper caches), and the engine logs a line saying so. Our setup recommends --remote-expert-opt for 2+ GPUs, so those two flags currently work against each other. --adapt-async 1 and --pipeline-windows also exclude each other. Please try, one at a time, "layer_split": "auto" with "gpu": [2, 0] (the 5060 Ti first or last, whichever is the faster card in the last stage), then --pipeline-windows 2 without --remote-expert-opt, and post the engine log lines about pipeline-windows and the layer split. The decode gap to the fork should shrink; we do not yet have a measured claim for your pair of cards.

  4. ztsvvstz1337 commented on Oct 8, 2026

    @ztsvvstz1337
    ztsvvstz1337
    on Oct 8, 2026

    Thanks for the numbers, and @ztsvvstz1337 for the PCIe notes. Here is what is in v0.1.40.x from that fork and what is not, from the commit log.

    In (with Francesco Albano as the author): the resident RAM mode on a layer split (fe9c10c), --vram-reserve-later-mib (5c4105c), the asynchronous adaptive expert tier --adapt-async 1 (b437b8e), the MTP drafter chain as one launch (8ba583f), pipelined verify windows --pipeline-windows (bfa532e), pipeline-windows beside the async tier (13eee45, #905, in 0.1.40.2), the CPU-side kernels (AVX2 Q8_K activation quantizer, AVX-VNNI forms, the gathered i-quant decode), and each card keeping only its own layers' weights (--trim-stage-weights, which also works under auto).

    Not the same as in the fork: in the fork several of these are on by default for two GPUs; here they are opt-in (--adapt-async 1, --pipeline-windows 2). A merge of the code is not a merge of the defaults, so the same model on the same cards does not run the same path unless the flags are given. Also not taken over: the automatic card order (the fork puts the faster card last in the split when layer_split is auto; the engine here keeps your order, which would explain the big gap between [0, 2] and [2, 0] on your prompts: the 4070 Ti SUPER should probably be the last stage), the fork's hybrid-CPU worker default (here setup only recommends it), the display-aware default VRAM reserves, and the "fewer host waits" commits for the window (we have not found a matching commit).

    One thing in your config that matters: --pipeline-windows is switched off by --remote-expert-opt (the helper caches), and the engine logs a line saying so. Our setup recommends --remote-expert-opt for 2+ GPUs, so those two flags currently work against each other. --adapt-async 1 and --pipeline-windows also exclude each other. Please try, one at a time, "layer_split": "auto" with "gpu": [2, 0] (the 5060 Ti first or last, whichever is the faster card in the last stage), then --pipeline-windows 2 without --remote-expert-opt, and post the engine log lines about pipeline-windows and the layer split. The decode gap to the fork should shrink; we do not yet have a measured claim for your pair of cards.

    quick follow up question or feature request (if this is not already implemented and/or done by the flags given above), is there an alternative way for multi gpu splitting, in which the card with the strongest pcie connection does the pre-filling only, and the other cards do text generation only? :)

  5. ztsvvstz1337 commented on Oct 8, 2026

    @ztsvvstz1337
    ztsvvstz1337
    on Oct 8, 2026

    Hi I might have some insight on the prompt processing. Its heavily PCIe lane speed dependent for example on my machine:
    x16 -> Transfers around 26gb/s -> 2000-3000 tps x4 -> ~ 5gb or less ~1000tps x2 -> super slow 1-2gb/s -> only 200-500 TPS

    I have a 4070 on 4.0 x16, and v100 on 3.0 x4 and I get around 100-150 prefill

    Thats kinda low honestly
    Maybe try a single card?
    E.g. I have 3x rtx 3090 with ~110 GB RAM
    I noticed with dual GPU ( x4 + x2 ) I get ~50-60 TPS with ~500 prefill but better batching and multiple requests
    But on the single GPU on x16 I get 80 TPS with 2500 prefill but its better for a single request

    All three cards together seems to slow down everything (might be cuz of GPU order didnt do much tests)

    Ive settled with a 256k context dual gpu multi agent API and a single 512k context high speed lane on the single 3090 which works well for me
    Numbers are eye balled averages not truly measured ;p

  6. Niko1221
    added 3 commits that reference this issue on Oct 8, 2026

    serve: card order for an auto layer split, faster card last (SMs x cl…

    be8db78

    setup: no --remote-expert-opt beside --pipeline-windows (the first tu…

    804dc7c

    setup: --remote-expert-opt only beside a helper cache; docs: --pipeli…

    54d2658
  7. Momoyeyu
    added a commit that references this issue on Oct 8, 2026

    bench: record dual-GPU auto-layout measurements for Niko1221#1352

    76edee7

  8. chemical12 commented on Oct 8, 2026

    @chemical12
    chemical12
    on Oct 8, 2026
    Author

    One thing in your config that matters: --pipeline-windows is switched off by --remote-expert-opt (the helper caches), and the engine logs a line saying so. Our setup recommends --remote-expert-opt for 2+ GPUs, so those two flags currently work against each other. --adapt-async 1 and --pipeline-windows also exclude each other. Please try, one at a time, "layer_split": "auto" with "gpu": [2, 0] (the 5060 Ti first or last, whichever is the faster card in the last stage), then --pipeline-windows 2 without --remote-expert-opt, and post the engine log lines about pipeline-windows and the layer split. The decode gap to the fork should shrink; we do not yet have a measured claim for your pair of cards.

    Thanks. I ran both tests on the same hardware and model with 0.1.41 version. The performance is now on par with the Hardin22 fork

    Hardware / environment

    • OS: Debian linux
    • CPU: Intel Xeon E5-2667 v4
    • RAM: 128 GB
    • PCI-E 3.0 x8
    • GPU [2]: RTX 4070 Ti SUPER 16 GB
    • GPU [0]: RTX 5060 Ti 16 GB
    • Version: 0.1.41
    • Model: Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S
    • GPUs: [2, 0]
    • layer_split: auto

    Test 1 — layer_split: auto

    Configuration:

    "gpu": [2, 0],
"layer_split": "auto"

    Docker console output

    2026-10-08T15:21:54.412527604Z [strata] layer split across GPUs [2, 0] (auto)
2026-10-08T15:21:54.412988711Z [strata] starting the engine: reading the model's weights ...
2026-10-08T15:22:03.918564654Z [strata] loading the experts into RAM (about 55 GB) and locking part of them for the GPU.
2026-10-08T15:22:03.918667345Z          YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
2026-10-08T15:22:03.918684495Z          Please wait and don't close this window; the browser opens when it is ready.
2026-10-08T15:22:23.932310899Z [strata] still starting (30 s) - please wait ...
2026-10-08T15:22:24.432506458Z [strata] experts loaded: 46.84 GiB at 3.94 GiB/s (30 s so far)
2026-10-08T15:22:24.932908381Z [strata] filling the GPU's expert cache (5854 experts, 10.52 GiB of VRAM) ...
2026-10-08T15:22:28.434811311Z [strata] almost ready ...
2026-10-08T15:22:31.175286417Z ready: http://127.0.0.1:8080/v1  (OpenAI: /v1/chat/completions, Anthropic: /v1/messages, context 262144 tokens, images on)
2026-10-08T15:22:31.175400274Z        open http://127.0.0.1:8080/ in a browser to chat; close this window to stop the model
2026-10-08T15:22:31.175419324Z        from other devices: http://172.16.19.2:8080/   (API: http://172.16.19.2:8080/v1)
2026-10-08T15:22:31.175431527Z        WARNING: no API key - anyone on your network can use this model. Add "api_key": "..." to the config (clients send it as their API key; the web page asks for it)
2026-10-08T15:23:07.955672662Z [strata] reading the prompt: 6,369 of 7,403 tokens, 8 s so far
2026-10-08T15:23:14.558417632Z [strata] reading the prompt: 7,398 of 7,403 tokens, 15 s so far
2026-10-08T15:23:15.715956775Z [strata] reading the prompt: 7,402 of 7,403 tokens, 16 s so far
2026-10-08T15:23:16.741903846Z [strata] thinking: 50 of max 254733 tokens, 51.2 tok/s, 17 s
2026-10-08T15:23:17.756186982Z [strata] thinking: 108 of max 254733 tokens, 54.2 tok/s, 18 s
2026-10-08T15:23:18.769245210Z [strata] thinking: 178 of max 254733 tokens, 59.3 tok/s, 19 s
2026-10-08T15:23:19.776967678Z [strata] thinking: 245 of max 254733 tokens, 61.1 tok/s, 20 s
2026-10-08T15:23:20.791511994Z [strata] thinking: 312 of max 254733 tokens, 62.1 tok/s, 21 s
2026-10-08T15:23:21.801867719Z [strata] thinking: 392 of max 254733 tokens, 64.9 tok/s, 22 s
2026-10-08T15:23:22.812313782Z [strata] thinking: 462 of max 254733 tokens, 65.6 tok/s, 23 s
2026-10-08T15:23:23.831780826Z [strata] thinking: 535 of max 254733 tokens, 66.3 tok/s, 24 s
2026-10-08T15:23:24.832718380Z [strata] thinking: 619 of max 254733 tokens, 68.3 tok/s, 25 s
2026-10-08T15:23:25.839511061Z [strata] thinking: 686 of max 254733 tokens, 68.1 tok/s, 26 s
2026-10-08T15:23:26.844620082Z [strata] thinking: 758 of max 254733 tokens, 68.4 tok/s, 27 s
2026-10-08T15:23:27.849521939Z [strata] thinking: 826 of max 254733 tokens, 68.4 tok/s, 28 s
2026-10-08T15:23:28.896233472Z [strata] thinking: 906 of max 254733 tokens, 69.0 tok/s, 29 s
2026-10-08T15:23:29.908760194Z [strata] thinking: 970 of max 254733 tokens, 68.6 tok/s, 30 s

    Engine log

    load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842

strata-vision: model files loaded in 1.2 s
strata-vision: on the CPU, 8 threads, no warm-up
[strata] 2026-10-08 15:21:54 engine started: /opt/strata/engine/strata --serve --pack /opt/Strata-data/packs/iq3_s --native /opt/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf /opt/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile /opt/strata/data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp /opt/Strata-data/mtp/rt --max-context 262144 --kv int8 --kv-resident 32768 --vision --vram-reserve-mib 700 --layer-split auto
strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)
strata generate: PCIe probe: 6.2 GB/s host->device (best of 6.2 5.9 6.0 6.0) -> pcie_frac 0.17 (default 0.55)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: /opt/Strata-data/packs/iq3_s experts (largest blob 2.66 MB), token embedding IQ4_XS in mapped host memory (322 MiB, 0.7 s)
strata generate: 1466 MiB of weights loaded from /opt/Strata-data/packs/iq3_s in 0.8 s (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 2018.88 MiB of weights, in 3.7 s
strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off)
strata generate: layer split: CUDA1 PCIe probe 6.0 GB/s (best of 6.0 6.0 6.0 6.0) -> pcie_frac 0.17
strata generate: layer split: CUDA1 holds its weights; 11.68 GiB free (its session follows the split search)
strata generate: profile /opt/strata/data/expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: layer split auto: CUDA0 36 SMs at 2.57 GHz -> 0.78 ms per layer, 10.99 GiB free before its session carve
strata generate: layer split auto: CUDA1 66 SMs at 2.67 GHz -> 0.41 ms per layer, 9.93 GiB free before its session carve
strata generate: layer split auto: CUDA0 link 6.7 GB/s -> miss cost x3.00
strata generate: layer split auto: CUDA1 link 6.0 GB/s -> miss cost x3.32
strata generate: layer split auto: K=19 - predicted 43.0 ms per decode window; the caches hold 10619 of 24576 profiled pairs (~97.4% of the routed mass), best of 46 placements
strata generate: KV streaming: 32768 of 262144 cells per QSA layer in VRAM, the K/V in 1.03 GiB of pinned RAM
strata generate: PLE on, table 320001536 rows of /opt/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf
strata generate: layer split: CUDA1 holds its weights, session [19, 48) and the head; 10.54 GiB free
strata mtp: draft layer loaded, 835 MiB of VRAM (experts 675, dense 111), files read in 0.78 s (1011 MiB/s)
strata generate: GPU 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0
strata generate: expert arena read through the file cache (buffered (not Windows))
strata generate: expert arena: cudaHostRegister PORTABLE ok; MAP_HUGETLB unavailable (needed 23983 2 MiB pages, vm.nr_hugepages=0); transparent huge pages are on for every mapping (THP always): MADV_HUGEPAGE not requested
strata generate: loaded 46.84 GiB at 3.94 GiB/s
strata generate: CPU pool tasks/phase: 24 (automatic), participating threads: 8
strata generate: 7 pool workers on logical processors 1,2,3,4,5,6,7, the host thread on 0 (draining too) (--host-core first)
strata generate: expert cache auto: 11.20 GiB free, 700 MiB reserved (+0 MiB for the draft head) -> 4242 slots
strata generate: expert cache 5854 slots, 10.52 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
                 so a reply can differ slightly from a run without the cache (same quality:
                 bench/results/2026-09-27-cache-parity).
                 PROFILE, ranked by routing frequency, no eviction.
strata generate: pre-filled 5854 of 5854 slots from the profile in 1.8 s (6405 MB/s); slot 0 verified
strata generate: layer split, CUDA1: 9.63 GiB free of 15.57, room for experts 8.85 GiB
strata generate: layer split: CUDA1 runs layers 19-47, expert cache 4486 slots (8.85 GiB), 4486 of its 14848 profiled pairs; slot 0 verified
strata generate: layer split: CUDA0 runs layers 0-18
strata generate: layer split auto: L2 probe: the split is startable on the live free figures
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 7 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.41)
strata generate: token graph hit path: 10340 resident experts, decided on the device
strata serve: prompt chunk auto: 8192 tokens, a 384-slot ring
strata serve: the prompt path borrows 2428 CUDA0 cache slots (4.38 GiB)
strata serve:   CUDA1 prompt path borrows 2223 of its 4486 slots (4.38 GiB)
strata hc: CUDA1: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata verify: window up to 6 tokens, 75.8 MiB of device buffers
strata serve: layer split: layers 0-18 (CUDA0), 19-47 (CUDA1), one hand-off per window
strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata verify: window up to 6 tokens, 75.8 MiB of device buffers
strata mtp: draft head over 106299 tokens (212.9 MiB)
strata serve: 390 MiB of VRAM free with everything loaded
strata verify: capturing the 4-token window (350 MiB of VRAM free)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: capturing the 4-token window (262 MiB of VRAM free)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: capturing the 1-token window (346 MiB of VRAM free)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: capturing the 1-token window (250 MiB of VRAM free)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: capturing the 2-token window (344 MiB of VRAM free)
strata verify: captured the 2-token window (upload no error, sync no error)
strata verify: capturing the 2-token window (234 MiB of VRAM free)
strata verify: captured the 2-token window (upload no error, sync no error)
strata verify: capturing the 3-token window (340 MiB of VRAM free)
strata verify: captured the 3-token window (upload no error, sync no error)
strata verify: capturing the 3-token window (222 MiB of VRAM free)
strata verify: captured the 3-token window (upload no error, sync no error)
strata verify: capturing the 5-token window (334 MiB of VRAM free)
strata verify: captured the 5-token window (upload no error, sync no error)
strata verify: capturing the 5-token window (208 MiB of VRAM free)
strata verify: captured the 5-token window (upload no error, sync no error)
strata serve: prompt 7403 tokens = 0 reused + 7403 read in 16237 ms (455.9 tok/s), 4247 generated in 65588 ms (64.8 tok/s), drafts accepted 1976 of 3812, 2 checkpoints
strata serve: decode expert cache hit rate: 94.2% (2745873 hits / 2915313 lookups); 5487 more read by the GPU over PCIe or from another GPU (0.2% of all 2920800 routed)
strata serve: KV streaming: 99.91% of 12506776 block reads hit VRAM, 43.5 MiB read from RAM
strata serve: suffix drafts: 85 windows, 134 of 189 drafts accepted

    Test 2 — --pipeline-windows 2

    Same GPU order and layer split:

    --pipeline-windows 2
"gpu": [2, 0],
"layer_split": "auto"

    Docker console output

    2026-10-08T15:32:52.488096025Z [strata] layer split across GPUs [2, 0] (auto)
2026-10-08T15:32:52.488532039Z [strata] starting the engine: reading the model's weights ...
2026-10-08T15:33:01.993852632Z [strata] loading the experts into RAM (about 55 GB) and locking part of them for the GPU.
2026-10-08T15:33:01.993940874Z          YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
2026-10-08T15:33:01.993958846Z          Please wait and don't close this window; the browser opens when it is ready.
2026-10-08T15:33:22.004532855Z [strata] still starting (30 s) - please wait ...
2026-10-08T15:33:28.008065855Z [strata] experts loaded: 46.84 GiB at 2.81 GiB/s (36 s so far)
2026-10-08T15:33:28.008176626Z [strata] filling the GPU's expert cache (5885 experts, 10.30 GiB of VRAM) ...
2026-10-08T15:33:31.009882789Z [strata] almost ready ...
2026-10-08T15:33:34.151311850Z ready: http://127.0.0.1:8080/v1  (OpenAI: /v1/chat/completions, Anthropic: /v1/messages, context 262144 tokens, images on)
2026-10-08T15:33:34.151403515Z        open http://127.0.0.1:8080/ in a browser to chat; close this window to stop the model
2026-10-08T15:33:34.151416502Z        from other devices: http://172.16.19.2:8080/   (API: http://172.16.19.2:8080/v1)
2026-10-08T15:33:34.151423302Z        WARNING: no API key - anyone on your network can use this model. Add "api_key": "..." to the config (clients send it as their API key; the web page asks for it)
2026-10-08T15:33:51.289763102Z [strata] reading the prompt: 6,369 of 7,403 tokens, 9 s so far
2026-10-08T15:33:58.047316027Z [strata] reading the prompt: 7,398 of 7,403 tokens, 15 s so far
2026-10-08T15:33:59.403980142Z [strata] reading the prompt: 7,402 of 7,403 tokens, 17 s so far
2026-10-08T15:34:00.404505364Z [strata] thinking: 56 of max 254733 tokens, 57.6 tok/s, 18 s
2026-10-08T15:34:01.413627997Z [strata] thinking: 128 of max 254733 tokens, 64.6 tok/s, 19 s
2026-10-08T15:34:02.420911971Z [strata] thinking: 202 of max 254733 tokens, 67.6 tok/s, 20 s
2026-10-08T15:34:03.436201544Z [strata] thinking: 290 of max 254733 tokens, 72.4 tok/s, 21 s
2026-10-08T15:34:04.447471639Z [strata] thinking: 365 of max 254733 tokens, 72.8 tok/s, 22 s
2026-10-08T15:34:05.463777694Z [strata] thinking: 459 of max 254733 tokens, 76.1 tok/s, 23 s
2026-10-08T15:34:06.487729901Z [strata] thinking: 522 of max 254733 tokens, 74.0 tok/s, 24 s
2026-10-08T15:34:07.506935586Z [strata] thinking: 596 of max 254733 tokens, 73.8 tok/s, 25 s
2026-10-08T15:34:08.542578958Z [strata] thinking: 664 of max 254733 tokens, 72.9 tok/s, 26 s
2026-10-08T15:34:09.560701197Z [strata] thinking: 735 of max 254733 tokens, 72.6 tok/s, 27 s
2026-10-08T15:34:10.563448375Z [strata] thinking: 812 of max 254733 tokens, 72.9 tok/s, 28 s
2026-10-08T15:34:11.582566740Z [strata] thinking: 885 of max 254733 tokens, 72.8 tok/s, 29 s
2026-10-08T15:34:12.583584367Z [strata] thinking: 964 of max 254733 tokens, 73.3 tok/s, 30 s

    Engine log — pipeline/layer split lines

    load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842

strata-vision: model files loaded in 1.1 s
strata-vision: on the CPU, 8 threads, no warm-up
[strata] 2026-10-08 15:32:52 engine started: /opt/strata/engine/strata --serve --pack /opt/Strata-data/packs/iq3_s --native /opt/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf /opt/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile /opt/strata/data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp /opt/Strata-data/mtp/rt --max-context 262144 --kv int8 --kv-resident 32768 --vision --vram-reserve-mib 700 --pipeline-windows 2 --layer-split auto
strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)
strata generate: PCIe probe: 6.0 GB/s host->device (best of 6.0 5.9 5.8 5.9) -> pcie_frac 0.16 (default 0.55)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: /opt/Strata-data/packs/iq3_s experts (largest blob 2.66 MB), token embedding IQ4_XS in mapped host memory (322 MiB, 0.6 s)
strata generate: 1466 MiB of weights loaded from /opt/Strata-data/packs/iq3_s in 1.0 s (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 2018.88 MiB of weights, in 3.1 s
strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off)
strata generate: layer split: CUDA1 PCIe probe 6.0 GB/s (best of 6.0 6.0 6.0 6.0) -> pcie_frac 0.17
strata generate: layer split: CUDA1 holds its weights; 11.68 GiB free (its session follows the split search)
strata generate: profile /opt/strata/data/expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: layer split auto: CUDA0 36 SMs at 2.57 GHz -> 0.78 ms per layer, 10.74 GiB free before its session carve
strata generate: layer split auto: CUDA1 66 SMs at 2.67 GHz -> 0.41 ms per layer, 9.77 GiB free before its session carve
strata generate: layer split auto: CUDA0 link 6.6 GB/s -> miss cost x3.04
strata generate: layer split auto: CUDA1 link 6.0 GB/s -> miss cost x3.32
strata generate: layer split auto: K=16 - predicted 43.4 ms per decode window; the caches hold 10526 of 24576 profiled pairs (~97.2% of the routed mass), best of 46 placements
strata generate: KV streaming: 32768 of 262144 cells per QSA layer in VRAM, the K/V in 1.03 GiB of pinned RAM
strata generate: PLE on, table 320001536 rows of /opt/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf
strata generate: layer split: CUDA1 holds its weights, session [16, 48) and the head; 10.53 GiB free
strata mtp: draft layer loaded, 837 MiB of VRAM (experts 675, dense 111), files read in 0.60 s (1320 MiB/s)
strata generate: GPU 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0
strata generate: expert arena read through the file cache (buffered (not Windows))
strata generate: expert arena: cudaHostRegister PORTABLE ok; MAP_HUGETLB unavailable (needed 23983 2 MiB pages, vm.nr_hugepages=0); transparent huge pages are on for every mapping (THP always): MADV_HUGEPAGE not requested
strata generate: loaded 46.84 GiB at 2.81 GiB/s
strata generate: CPU pool tasks/phase: 24 (automatic), participating threads: 8
strata generate: 7 pool workers on logical processors 1,2,3,4,5,6,7, the host thread on 0 (draining too) (--host-core first)
strata generate: --pipeline-windows 2: 234 MiB of CUDA0 kept out of the expert cache (the second verifier, the GDN snapshots)
strata generate: expert cache auto: 11.21 GiB free, 700 MiB reserved (+0 MiB for the draft head) -> 4153 slots
strata generate: expert cache 5885 slots, 10.30 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
                 so a reply can differ slightly from a run without the cache (same quality:
                 bench/results/2026-09-27-cache-parity).
                 PROFILE, ranked by routing frequency, no eviction.
strata generate: pre-filled 5885 of 5885 slots from the profile in 1.7 s (6526 MB/s); slot 0 verified
strata generate: layer split, CUDA1: 9.62 GiB free of 15.57, room for experts 8.69 GiB
strata generate: layer split: CUDA1 runs layers 16-47, expert cache 4379 slots (8.69 GiB), 4379 of its 16384 profiled pairs; slot 0 verified
strata generate: layer split: CUDA0 runs layers 0-15
strata generate: layer split auto: L2 probe: the split is startable on the live free figures
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 7 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.41)
strata generate: token graph hit path: 10264 resident experts, decided on the device
strata serve: prompt chunk auto: 8192 tokens, a 384-slot ring
strata serve: the prompt path borrows 2489 CUDA0 cache slots (4.38 GiB)
strata serve:   CUDA1 prompt path borrows 2208 of its 4379 slots (4.38 GiB)
strata hc: CUDA1: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata verify: window up to 6 tokens, 75.8 MiB of device buffers
strata serve: layer split: layers 0-15 (CUDA0), 16-47 (CUDA1), one hand-off per window
strata verify: window up to 6 tokens, 75.8 MiB of device buffers
strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata verify: window up to 6 tokens, 75.8 MiB of device buffers
strata verify: window up to 6 tokens, 75.8 MiB of device buffers
strata mtp: draft head over 106299 tokens (212.9 MiB)
strata serve: --pipeline-windows 2: two verifiers per stage, the stages overlap across windows (decode too; the GDN snapshots beside the window)
strata serve: 472 MiB of VRAM free with everything loaded
strata verify: capturing the 1-token window (432 MiB of VRAM free)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: capturing the 2-token window (432 MiB of VRAM free)
strata verify: captured the 2-token window (upload no error, sync no error)
strata verify: capturing the 3-token window (426 MiB of VRAM free)
strata verify: captured the 3-token window (upload no error, sync no error)
strata verify: capturing the 4-token window (424 MiB of VRAM free)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: capturing the 5-token window (420 MiB of VRAM free)
strata verify: captured the 5-token window (upload no error, sync no error)
strata verify: capturing the 6-token window (416 MiB of VRAM free)
strata verify: captured the 6-token window (upload no error, sync no error)
strata verify: capturing the 1-token window (410 MiB of VRAM free)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: capturing the 2-token window (406 MiB of VRAM free)
strata verify: captured the 2-token window (upload no error, sync no error)
strata verify: capturing the 3-token window (402 MiB of VRAM free)
strata verify: captured the 3-token window (upload no error, sync no error)
strata verify: capturing the 4-token window (396 MiB of VRAM free)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: capturing the 5-token window (394 MiB of VRAM free)
strata verify: captured the 5-token window (upload no error, sync no error)
strata verify: capturing the 6-token window (388 MiB of VRAM free)
strata verify: captured the 6-token window (upload no error, sync no error)
strata verify: capturing the 1-token window (340 MiB of VRAM free)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: capturing the 2-token window (328 MiB of VRAM free)
strata verify: captured the 2-token window (upload no error, sync no error)
strata verify: capturing the 3-token window (314 MiB of VRAM free)
strata verify: captured the 3-token window (upload no error, sync no error)
strata verify: capturing the 4-token window (302 MiB of VRAM free)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: capturing the 5-token window (288 MiB of VRAM free)
strata verify: captured the 5-token window (upload no error, sync no error)
strata verify: capturing the 6-token window (276 MiB of VRAM free)
strata verify: captured the 6-token window (upload no error, sync no error)
strata verify: capturing the 1-token window (260 MiB of VRAM free)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: capturing the 2-token window (248 MiB of VRAM free)
strata verify: captured the 2-token window (upload no error, sync no error)
strata verify: capturing the 3-token window (234 MiB of VRAM free)
strata verify: captured the 3-token window (upload no error, sync no error)
strata verify: capturing the 4-token window (222 MiB of VRAM free)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: capturing the 5-token window (208 MiB of VRAM free)
strata verify: captured the 5-token window (upload no error, sync no error)
strata verify: capturing the 6-token window (194 MiB of VRAM free)
strata verify: captured the 6-token window (upload no error, sync no error)
strata serve: prompt 7403 tokens = 0 reused + 7403 read in 16701 ms (443.3 tok/s), 3646 generated in 52831 ms (69.0 tok/s), drafts accepted 1703 of 3253, 2 checkpoints
strata serve: decode expert cache hit rate: 93.4% (2732565 hits / 2926787 lookups); 8193 more read by the GPU over PCIe or from another GPU (0.3% of all 2934980 routed)
strata serve: KV streaming: 99.94% of 16339504 block reads hit VRAM, 41.2 MiB read from RAM
strata serve: suffix drafts: 32 windows, 55 of 82 drafts accepted

    With a big ~200k-token prompt, the prompt processing speed was approximately 2394.30 t/s tokens/s.

    The tests were performed with the same model, hardware, GPU order, and context settings. Only the specified configuration changes were made between the tests.

  9. fukc-gihtub commented on Oct 8, 2026

    @fukc-gihtub
    fukc-gihtub
    on Oct 8, 2026
    Contributor

    This felt like a regression. Prefill is 25-30% faster with "gpu_order": "as_given" on my machine (dual, asymmetrical Blackwell GPUs).

  10. forestautomation
    added a commit that references this issue on Oct 10, 2026

    Strata-Cascade: merge upstream 0.1.41, batched cold reads, and the tu…

    a293d8a
  11. Niko1221
    added a commit that references this issue on Oct 10, 2026

    serve: keep the given card order when the reorder starves the last st…

    efe3bfc

  12. MarsLuay commented on Oct 13, 2026

    @MarsLuay
    MarsLuay
    on Oct 13, 2026

    I ran the two tests requested in the maintainer reply on Strata v0.1.43.1 CUDA 12 with an RTX 3080 Ti 12 GB + Tesla V100 32 GB PCIe pair on Windows/WDDM. Model: Qwen3.8-Flash-Next GSQ-RCO IQ3_S, 131K context, --spec 4 --spec-min-p 0.5 --mtp, 10K prompts, 128 greedy output tokens, two measured requests per arm.

    Auto order with the larger/slower card first

    With gpu: [1, 0] (V100 first, RTX second) and layer_split: auto, v0.1.43.1 selected:

    layer split auto: K=26
layer split: layers 0-25 (CUDA0/V100), 26-47 (CUDA1/RTX)

    The V100 held 24.19 GiB of expert cache and the RTX stage held 3.81 GiB. Prompt: 1,029.9 / 1,041.2 tok/s (mean 1,035.6). Decode: 66.7 / 76.4 tok/s (mean 71.6).

    For comparison, gpu: [0, 1] auto selected K=24 with RTX layers 0-23 and V100 layers 24-47: prompt mean 1,155.7 tok/s, decode mean 60.9 tok/s. Thus the reversed order is slower for prompt ingestion but faster for decode on this PCIe pair; importantly, the RTX is the last stage while the V100 retains the larger cache.

    Pipeline without the helper-cache flag

    With manual K=28 and --trim-stage-weights --pipeline-windows 2, omitting --remote-expert-opt, startup reported:

    --pipeline-windows 2: two verifiers per stage, 2 stages overlap across windows
layer split: layers 0-27 (CUDA0/RTX), 28-47 (CUDA1/V100)

    Prompt: 1,445.7 / 1,532.8 tok/s (mean 1,489.2). Decode: 78.6 / 76.7 tok/s (mean 77.7). All verification captures reported upload/sync without error.

    Both tests ran successfully. These are short two-run measurements, but the order and pipeline behavior match the requested diagnostic flags and log lines.

  13. Niko1221 commented on Oct 13, 2026

    @Niko1221
    Niko1221
    on Oct 13, 2026
    Owner

    Thanks for the report. To keep the tracker current we are closing issues older than three days: 0.1.44 is out with a long list of fixes, and many older reports are covered by it or by 0.1.43.1.

    If this still happens on 0.1.44, please leave a comment here with your engine log (and the version line it prints), and we will reopen it. A new issue that links this one works too.

  14. Niko1221
    closed this as not plannedon Oct 13, 2026
  15. giridhart
    added 4 commits that reference this issue on Oct 15, 2026

    serve: card order for an auto layer split, faster card last (SMs x cl…

    3b20338

    setup: no --remote-expert-opt beside --pipeline-windows (the first tu…

    83d0ee4

    setup: --remote-expert-opt only beside a helper cache; docs: --pipeli…

    aafa4ca

    serve: keep the given card order when the reorder starves the last st…

    169cd9d
  16. DarthCY-K
    added a commit that references this issue on Oct 20, 2026

    server: a card on a slow PCIe link goes last in a layer split

    b1fdf0b
  17. Momoyeyu
    added a commit that references this issue on Oct 20, 2026

    bench: record dual-GPU auto-layout measurements for Niko1221#1352

    6b010de
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions