Skip to content

[Feature]: Layer-major prefill for streamed experts — 4–6× faster long prompts on a 6 GB card (PRs offered) #1670

Closed as not planned
Closed as not planned
[Feature]: Layer-major prefill for streamed experts — 4–6× faster long prompts on a 6 GB card (PRs offered)#1670

Description

@vizdatom
vizdatom
opened on Oct 9, 2026

What do you want to do?

Make prompt reading fast on a small-VRAM card when the experts are streamed (--mmap-experts). On a 6 GB RTX 3060 Laptop with Qwen3.8-Flash-Next Q2_0, a layer-major prefill takes long prompts from 62–99 tok/s to 378–443 tok/s (a 223.7K-token prompt from ~2,500 s to ~520 s), with decode about the same (19–21 tok/s). I have this working on a local branch and would like to open PRs if you're interested.

Why it is slow today

With --prefill auto on 6 GB, the prompt is read in 512-token chunks and every chunk walks all layers, so each layer's routed experts are streamed from the file tier once per chunk. On this box a 12.9K-token prompt moves ~509 GB of expert bytes, almost all of it the same experts re-read 25 times. #1407 looks like the same re-read pattern on a layer split ("file tier re-reads 24–35 GB per request").

What I changed (all opt-in env switches, default off)

  1. STRATA_PREFILL_LAYER_MAJOR=1 — layer-major prefill. A super-chunk of up to STRATA_PREFILL_LM_TOKENS (default 8192, shrunk to free VRAM minus STRATA_PREFILL_LM_MARGIN_MIB) is read layer by layer. Each 512-token sub-chunk runs attention, the hyper-connection read, the router and the shared expert in order (GDN / QSA / PLE state advances as in the default path). The layer's routed experts then run once for the whole super-chunk: each expert is staged once, its rows from every sub-chunk go into one MMQ group, and the results are added per token. The super-chunk's residuals live in VRAM or pinned RAM (STRATA_PREFILL_LM_R=vram|host|auto, copies overlapped on the copy stream). Every streamed expert of a layer is prefetched when its attention starts, and the ones the routing skips are dropped (STRATA_LM_PREFETCH=0 turns this off). Expert traffic drops by about S / 512: the 12.9K prompt goes from ~509 GB to ~79 GB.
    • Two small kernels (moe_shared_init, moe_scatter_add).
    • A stager fix the prefetch needs: a job now waits for the previous job on its pinned buffer, and skip() marks unneeded jobs ready without reading them. Before, two read threads could write the same buffer. The default walk consumes jobs in order, so it isn't affected.
    • on_chunk still reports every sub-chunk; serve doesn't checkpoint mid-super-chunk.
    • Not used for a layer split, a peer GPU, image prompts, the fused layout, or prompts under two chunks.
  2. STRATA_PREFILL_DIRECT=1 — split page cache vs O_DIRECT by residency. On the prompt path, each expert blob gets a mincore residency check. Mostly cached (≥ STRATA_PREFILL_DIRECT_MIN, default 50%) means memcpy from the mapping; otherwise it's read with O_DIRECT. Decode stays on the page cache. Prompts no longer evict the decode working set: prefill +3–27% and decode +11–17% on this box.
  3. STRATA_PREFILL_WARM=1 / STRATA_BIN_WARM=1 — the per-layer walk warms this and the next layer's file-tier experts, and the pack's experts.bin is warmed too (Linux). Small: +1–9% prefill.

Results

Ryzen 7 5800H, RTX 3060 Laptop 6 GB, 27 GiB RAM, Apacer AS2280P4 NVMe (Gen3, 256 GB), Ubuntu 24.04, driver 595.99.02.

Engine: 0.1.39 + 10 commits (idx-int8-524k), with your bit-plane Q2_0 kernel and Linux O_DIRECT tier commits cherry-picked.

Model: Qwen3.8-Flash-Next GSQ-RCO Q2_0 pack. Flags: --mmap-experts --expert-cache auto --prefill auto --kv q4_0 --kv-resident 32768 --max-context 262144 --spec 3 --spec-min-p 0.5 --mtp … --vram-reserve-mib 650.

Methodology: page cache evicted before each run, unique prefix per request (no reuse), temperature 0.6, thinking off; tok/s from the engine's own strata serve: prompt … log lines.

prompt 7.4K prompt 8.1K prompt 12.9K prompt 51.5K decode (prose / code)
0.1.39 base 69 74 95 – 17.6 / –
+ bit-plane, both warms 75 81 96 – 18.5 / 18.3
+ PREFILL_DIRECT 95 98 99 62.5 20.6 / 21.4
+ PREFILL_LAYER_MAJOR 378 406 443 415 19.1 / 19.8

223,706-token needle prompt: 2,400–2,611 s before, 498–538 s after (3/3 found).

Correctness with layer-major on, served for agent use:

  • 17-check tool-calling test (tool choice from a menu, nested/optional/enum params, multi-turn chains, no-tool and error recovery): 17/17
  • needle at 223.7K: 3/3
  • BFCL multi_turn_base: 55.5% (200 entries, 7 h 52 min)
  • engine errors: none across 2,547 requests

Side finding: with --resident-budget-gib 14 on 27 GiB RAM, the Linux O_DIRECT tier picks unbuffered reads by itself and decode falls from 18.4 to 4.5 tok/s, because decode misses go to the NVMe. STRATA_UNBUFFERED_LOAD=0 restores it. Prompt reads did get faster in that mode (117 vs 77 tok/s); item 2 above gets that gain without the decode loss.

Proposal

My branch is 0.1.39-based and main has moved a lot (#1107, #1247, #1441 also touch prefill). Before rebasing, I'd like to know if you want this and in what shape. My suggestion is three PRs against current main, each re-measured after the rebase:

  1. Layer-major prefill, with its kernels and the stager fix
  2. PREFILL_DIRECT
  3. The two warm switches

Happy to change names, defaults or structure to fit your plans for the prefill path.

Activity

  1. CC-David-CC commented on Oct 9, 2026

    @CC-David-CC
    CC-David-CC
    on Oct 9, 2026
    Contributor

    put your branch up on github and i'll test it. idc that its on 1.39 i have a private branch with some divergences i'm maintaining and i'd love to see how you got this in practice

  2. vizdatom commented on Oct 9, 2026

    @vizdatom
    vizdatom
    on Oct 9, 2026
    ContributorAuthor

    Thanks! It's up: https://github.com/vizdatom/Strata/tree/perf-ryzen

    Diff against its base: vizdatom/Strata@7026cc8...perf-ryzen

    Base: 0.1.39 + 10 commits (7026cc84, the idx-int8-524k branch).

    Cherry-picked from upstream main:

    • the bit-plane Q2_0 kernel: 845ac757, 97566cf3, 6316091a
    • Linux O_DIRECT reads: b3980df3

    The new work:

    • 6b275dc4: STRATA_BIN_WARM
    • 1cd6dac4: STRATA_PREFILL_WARM
    • bd841299, 1d943c06: STRATA_PREFILL_DIRECT
    • 2d8bd284: layer-major kernels
    • 00caa81d: stager wait/skip
    • 6453e8a7: STRATA_PREFILL_LAYER_MAJOR
    • 3720d627: debug dumps only (STRATA_DUMP_*), used for the layer-major A/B

    The environment I run it with (engine flags as in the issue):

    STRATA_Q2_BITPLANE=1
STRATA_BIN_WARM=1
STRATA_PREFILL_WARM=1
STRATA_PREFILL_DIRECT=1
STRATA_PREFILL_LAYER_MAJOR=1
STRATA_LOOKAHEAD=0
STRATA_STAGER_THREADS=16
STRATA_STAGER_RING=256

    Layer-major only applies with streamed experts on a single GPU and prompts of two or more chunks. The engine log line strata serve: prompt N tokens = … read in … ms (… tok/s) is where the gains show. Curious how it behaves on your setup.

  3. Niko1221 commented on Oct 13, 2026

    @Niko1221
    Niko1221
    on Oct 13, 2026
    Owner

    Thanks for the report. To keep the tracker current we are closing issues older than three days: 0.1.44 is out with a long list of fixes, and many older reports are covered by it or by 0.1.43.1.

    If this still happens on 0.1.44, please leave a comment here with your engine log (and the version line it prints), and we will reopen it. A new issue that links this one works too.

  4. Niko1221
    closed this as not plannedon Oct 13, 2026

  5. willthehuman commented on Oct 14, 2026

    @willthehuman
    willthehuman
    on Oct 14, 2026
    Contributor

    Still reproducing on 0.1.44, so per your comment above here is the log and the version line.

    Version line the engine prints: strata 0.1.44.1

    Setup: i7-8700, 31.9 GB RAM, RTX 4070 SUPER 12 GB, Q2_0, --mmap-experts --resident-budget-gib 16 --max-context 131072, storage on ntfs3 (DRAM-less NVMe).

    What the prompt path still does here, from the engine's own per-request line, for a single 17,280-token prompt:

    strata serve: file tier I/O this request: the OS read 66117.1 MB from storage (45079 major faults) for 44388.9 MB of expert reads; page cache held 36173.8 MB at the read, not 25394.2 MB

    44.2 GB of expert bytes for one 17K-token prompt, with the OS delivering 66 GB and 45,079 major faults; prefill 442.4 tok/s on that request. The same prompt with STRATA_UNBUFFERED_LOAD=1 reads the same bytes at 343 faults and 630.0 tok/s:

    strata serve: file tier I/O this request: the OS read 66413.0 MB from storage (343 major faults) for 44312.8 MB of expert reads; page cache held 0.0 MB at the read, not 0.0 MB

    So on 0.1.44.1 the read shape still dominates the prompt path — the amplification this issue describes is unchanged, and a 12 GB card here streams every layer's experts per chunk the same way you measured on the 6 GB one. Changing the read path recovers 43% of the prefill; it does not reduce the 44 GB of expert traffic, which is the part your layer-major proposal targets.

    I do not have the layer-major path to test on 0.1.44.1 (vizdatom's perf-ryzen branch is 0.1.39 + 10 commits). Would you reopen this? I am happy to test a rebased branch or an upstream implementation and report numbers with the same protocol — cold cache verified, identical prompt, interleaved arms — plus whatever logs you want.

  6. vizdatom commented on Oct 15, 2026

    @vizdatom
    vizdatom
    on Oct 15, 2026
    ContributorAuthor

    Hi @willthehuman, thanks for re-measuring on 0.1.44.1, that's exactly the case the issue is about. On small cards, layer-major saves a lot and on cards that don't stream experts, it costs almost nothing.

    I've pushed my port onto 0.1.43.1: https://github.com/vizdatom/Strata/tree/layer-major-0.1.43.1 (diff: vizdatom/Strata@a0116b2...layer-major-0.1.43.1). It uses the same env switches as in the issue. On my 6 GB card it needs --expert-cache 4 --prefill 512 --no-prefill-borrow, because with auto the engine exits at startup.

    Feel free to take it from here and do whatever you want with it. My hardware is busy with another project now, so a rebase onto 0.1.44 and re-testing aren't in my near-term plans.

  7. solomon159753-oss commented on Oct 18, 2026

    @solomon159753-oss
    solomon159753-oss
    on Oct 18, 2026

    Superseded by #1670 (comment) after moving this report to my durikim1-cpu account. Please use the replacement comment for the measurements and experiment branch.

  8. durikim1-cpu commented on Oct 18, 2026

    @durikim1-cpu
    durikim1-cpu
    on Oct 18, 2026

    Data point from HIP on Windows (AMD RX 7800 XT 16 GB, gfx1101) on a 0.1.44.1 base (the thread so far has a CUDA one on 0.1.44.1): it still matters there too.

    Credit first: the layer-major code is @vizdatom's, from his layer-major-0.1.43.1 branch
    (2e69b6db "prefill: layer-major prefill (STRATA_PREFILL_LAYER_MAJOR=1)", f0173382 kernels for the layer-major
    expert pass, 149b8995 the stager wait, 7013dad4/bd900165 the prompt-path direct reads, 7753c2e2 the debug
    dumps). I ported it onto my 0.1.44.1-based tree and ran it on HIP/Windows; I did not design it.

    Setup

    • RX 7800 XT 16 GB (gfx1101), Ryzen 5 9600X, 64 GB DDR5, NVMe, Windows 11, HIP built with the ROCm SDK 10.2.0a20260930 (runtime amdhip64_7.dll 10.0.3686.0, hipBLASLt 1.5.0).
    • Flash-Next GSQ-RCO IQ3_XXS native pack, 64K context, int8 KV, MTP on (spec 4), Thinking off, temperature 0,
      expert RAM budget 20 GiB, STRATA_PREFILL_LAYER_MAJOR=1 with a 16,384-token super-chunk.
    • Both arms are the same private build lineage: 0.1.44.1 plus my own earlier local changes (a compact MMQ scratch
      buffer and a few existing opt-ins, among them the gfx1101 hipBLASLt table). The candidate adds the port and also carries libratechw's opt-in HIP prompt-attention patches (STRATA_ATTN_GATE, STRATA_QPREP, not enabled in these runs). It is not a measurement of 0.1.44.1 as released.
    • Fresh 60,047-token prompt (0 reused tokens), output cap 384, three runs per arm, engine timings for tok/s, client
      timing for TTFT. Control, then candidate, then the control again on the same day.

    Results (medians of 3)

    Arm prompt tok/s decode tok/s TTFT s
    control (before) 896.6 26.0 67.06
    layer-major 16K 1,177.4 (+31.3%) 27.0 51.10 (−23.8%)
    control (after) 901.4 27.5 66.71

    The control measured again afterwards stayed at 901.4 tok/s, so the gain is not drift. Decode did not get faster
    (27.0 sits between the two controls), as on @vizdatom's RTX 3060 Laptop (6 GB) numbers.

    Per request, the expert file source's logical reads fell from about 171 GB to 82 GB, average GPU busy rose from
    71.7% to 84.5%, and observed physical SSD reads fell from 2.46 to 1.46 GB/s (whole drive). "Logical" is what the
    engine asked for, not what came off the SSD.

    Correctness on the layer-major arm: a 12-question set 11/12 (the same arithmetic question fails without it),
    long-context retrieval 12/12, and a real coding agent (6 tasks × 3) 18/18 functional (15/18 on a stricter
    process check).

    On top of it (my own, small)

    shared-arena-guard.patch (112 added lines in prefill.cpp, both opt-in): STRATA_PREFILL_LM_CACHE_ARENA=1 lets the
    layer-major pass borrow its storage from the expert cache through the existing lend/refill protocol and return it
    before decoding, instead of a separate allocation; STRATA_PREFILL_LM_CACHE_GUARD=1 charges that borrowed capacity
    once and keeps the original free-VRAM margin. With a 32,768-token super-chunk and a 23 GiB budget this reached
    1,408.9 prompt tok/s / 27.5 decode / 42.70 s TTFT on the same 60K prompt. A 32K super-chunk with a separately
    preallocated buffer (20 GiB budget) reached 1,416.1 prompt tok/s but only 20.7 decode. These two rows differ in more
    than one setting, so read them as a direction, not an A/B.

    Not tested

    The port on a clean 0.1.44.1 or current main, CUDA or Linux builds of the port, contexts above 64K with these
    settings except one 100K check at a 128K context and a 22 GiB budget (100,055 tokens: 1,185.1 prompt tok/s, 23.9 decode, 3/3 retrieval), and any other GPU.

    Port and the extra patch, for anyone who wants to try it on HIP/Windows: https://github.com/durikim1-cpu/Strata/tree/rx7800xt-experiments
    (layer-major-port-0.1.44.1-hip.patch is against my tree, not against main).

    If you would consider reopening this for an opt-in layer-major path, I can re-measure the port on a clean build.

  9. Niko1221
    added 5 commits that reference this issue on Oct 27, 2026

    prefill stager: a job waits for the previous job on its buffer; skip() (

    fca1936

    prefill: the per-layer walk warms this layer's and the next layer's f…

    162dec3

    prefill: kernels for a layer-major expert pass (moe_shared_init, moe_…

    a6df002

    prefill: layer-major prefill (STRATA_PREFILL_LAYER_MAJOR=1) (#3976, #…

    ffe40c1

    layer-major prefill: STRATA_HC_Q8 dequant, guards for APR/NVFP4/WMMA …

    df7068b
  10. atmJava
    added a commit that references this issue on Oct 28, 2026

    prefill stager: a job waits for the previous job on its buffer; skip() (

    fbbfad8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions