[Hardin22]

This is the part of the fork's "fewer, fuller launches" work that main doesn't have yet. It comes as two commits, and each piece has its own switch. Every piece keeps the arithmetic of the path it replaces, so the output is unchanged; the checks are below.

1. Programmatic dependent launch on sm_90+ (STRATA_DF_PDL). A decode window is a long chain of small kernels, and on a fast card many of them finish before the next one has fetched its weights. With PDL, the quantize kernel, the multi-column MMVQ and the multi-row BF16 GEMV load their weights before griddepcontrol.wait.
   • Where it applies: the attribute is only set while capturing, and only when every predecessor of the next node is a kernel (never after a memcpy, memset, host or event node). STRATA_DF_PDL=2 restricts it to single-predecessor edges.
   • When it is on: pdl_supported() is false on HIP and on CUDART < 12.3. It also needs cc_major >= 9 and a binary built for sm_90+ (PTX JIT from older code would lack the wait).
   • Other changes:
     • copy_rows_strided replaces a memcpy2D node.
     • The three PDL kernels drop __restrict__ (llama.cpp's PDL fix), which can change speed but not values.
2. Graph branches and batched forms:
   • STRATA_DF_BRANCH: the GDN a/b/z, query and indexer-query sides and the step/pos inputs on a side branch. Opt-in since f89d31f: on Linux (open modules, GSP) it stalled with NVML queries between requests (Pipelined windows + async adaptive tier together (stacked on #859 and #876) #905).
   • STRATA_DF_QB_Q4, STRATA_DF_KVAPP: the Q4_0 KV's window batched (main still appends per token with --kv q4_0).
   • STRATA_DF_PLE: the PLE post-ops in one pass.

Already in main, so left out: native_swiglu_quantize_q8_1 / sigmoid_scale_rows, fd95405's attention change, the shared-expert branch, and STRATA_PLE_BATCH's projections. I also dropped a one-row-per-warp MMVQ variant, because it did not beat main's ROWS=2.

Measured

Swift 1.5 IQ3_XXS at 160K (q4_0 KV), RTX 4060 Ti + RTX 5080 layer split, with the resident RAM mode (#848), pipelined windows and the async tier (#859, #876). Greedy, five prompts × two rounds, decode tok/s.

	run 1	run 2
every switch on (default)	123.5	119.7
every switch off (STRATA_DF_PDL=0 STRATA_DF_BRANCH=0 STRATA_DF_QB_Q4=0 STRATA_DF_KVAPP=0 STRATA_DF_PLE=0)	118.1	117.5

About +3%. PDL only acts on the 5080 (sm_120); the 4060 Ti (sm_89) has none.

Checks

• Exact pass (STRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 0, serial loop, 4 prompts × 500 tokens): all switches on and all off give the same text.
  • Against a build without these commits, 2 of 4 differ. The larger binary leaves 2 fewer expert slots on each card (5740 vs 5742, 4762 vs 4764), so a few experts move to the CPU, which rounds them differently.
• Unit tests pass on both cards (RTX 5080, RTX 4060 Ti):
  • pdl_parity: 0 of 200 replays differ; programmatic edges only go kernel → kernel; also with STRATA_DF_PDL=2.
  • New: verify_batch_parity (every batched form against its per-token path, bitwise).
  • Existing: mmvq_multi_parity, kv_q4_parity, bf16_gemv_parity.

Not tested

• HIP and CUDA 12.3–12.9 builds were not compiled here.
  • HIP takes the plain launch.
  • griddepcontrol is only compiled under !__HIPCC__ && __CUDA_ARCH__ >= 900.

[blange48]

Feedback on a 4-way split (rig: 4 x RTX 5080 16 GB (PCIe Gen3 x16, sm_120), i9-10920X, Linux, IQ3_S, 262K context, int8 K/V, layer split 12,24,36 with --trim-stage-weights, --batch 8 --batch-groups 4, MTP spec 4): applied on top of 0.1.39 together with other patches, in production since today.

• Exact: tools/batch_test.py gives slots identical to solo with 8 and 3 slots (greedy, --pcie-frac 0 --adapt-every 1000000, STRATA_IQ_MT_MIN=1), and parking_test.py is identical.

• Speed through the HTTP server, all switches on (default) vs STRATA_DF_PDL=0 STRATA_DF_BRANCH=0 STRATA_DF_QB_Q4=0 STRATA_DF_KVAPP=0 STRATA_DF_PLE=0, total tok/s:

  Concurrent	off	on
  1	115.2	122.1
  2	151.6	150.6
  4	252.2	251.3
  8	367.1	372.1

So about +5 % for one request (the solo path with MTP drafts) and neutral to +1 % in batch windows, consistent with your +3 %. These are single runs; our run-to-run noise is about ±3 %.

• No NVRM/Xid messages in dmesg across the engine starts and stops.

[Hardin22]

Thank you for running it on 4x 5080 in production. It's useful to see slots identical to solo with batch 8, and +5% on single requests.

One change since then: STRATA_DF_BRANCH is now opt-in (f89d31f). On #905, adambenhassen bisected a stall on Linux (open modules 610.43.02, GSP) to the branches in the verify graph combined with an NVML query (nvidia-smi, any field) between requests:

• the window stops inside its graph at the GDN fork;
• 80/80 clean with STRATA_DF_BRANCH=0, 30/30 clean without the query;
• on 2x 3090 and on 2x 2080 Ti over NVLink;
• not seen on Windows.

PDL, the Q4_0 KV batching and the PLE post-ops keep their defaults. If your monitoring polls NVML while the engine serves, I'd turn the branches off until we know more.

[Niko1221]

Heads-up: today we cleaned up this repository's commit history, and GitHub closed this PR automatically when main was force-pushed (that is the "base branch was force-pushed" event above). It is not a rejection.

GitHub cannot reopen it, because its branch is built on the old history. If you want to keep it going: start a new branch from the new main, cherry-pick your own commits onto it (git fetch upstream && git switch -c my-fix upstream/main && git cherry-pick <your commits>), and open a new PR. Or simply leave it: when we take your change for a release we port it onto the new main ourselves and keep you as its author.

Sorry for the noise, and thank you for the contribution.

[Niko1221]

Thanks, Hardin22. The PDL, PLE post-op, Q4_0 query-path and graph-branch parts are in 0.1.40.2, credited as co-author, all opt-in (STRATA_DF_PDL, STRATA_DF_PLE, STRATA_DF_QB_Q4, STRATA_DF_BRANCH). The Q4_0 KV append part was already in 0.1.40 (kv_append_q4_steps), so it was dropped. Defaults stay off until a 10-pair A/B on sm_90+ hardware says otherwise.