[atomicmilkshake]

What this is

A stacked PR on top of #1402 (qwen35moe support). It contains the qwen35moe follow-up work we built
and measured while validating that PR on one machine (RTX 3080 10 GB, Windows 11, CUDA 13.2, sm_86):

• MTP for third-party single-file GGUFs whose draft block is stored as F16;
• an opt-in VRAM fraction cap (--vram-frac) and a quality cap mode (--vram-cap-mode quality)
  that keeps every expert miss on the same GPU kernel as a hit;
• opt-in chunked DeltaNet prefill for qwen35moe;
• route-resident routing generalized to 256-expert / top-8 models (the route-resident: one warp per token in the swap kernel #1737 warp-shuffle idea,
  merged with our 256/8 implementation);
• the determinism fixes the auto expert cache needed on Windows/WDDM;
• setup-family and SYCL follow-ups.

Base: #1402 head 48534f3. The first commits in the diff are #1402's; the 16 commits listed below are
this PR's content. The five upstream PRs we also adopted locally (#1700, #1660, #1659, #1696, #1720) are
deliberately not included — they are separate PRs and should land on their own.

The repo's guide is one change per pull request. This is a stack of related qwen35moe follow-ups; I am
happy to split it into separate PRs (MTP-F16; cap+quality; chunked GDN; route-resident 256/8) if that is
easier to review. The MTP F16 commit is the smallest and probably the most valuable on its own.

Commits

MTP for third-party GGUFs

• MTP: accept F16 draft tensors in a single-file GGUF — the draft-layer loader took F32 for norms and
  BF16/F32 for routers, so a single-file GGUF whose MTP block keeps ffn_gate_inp.weight at F16 refused to
  start. F16→F32 is exact; F16→BF16 uses the same round-to-nearest-even as the F32 path. The finite cases
  check out over all 65,536 half bit patterns; NaN needed a later fix (the first checker skipped NaNs, and
  the rounded addition carried them into infinity/signed zero — see the follow-up commit below). Measured on
  the Huihui Q4_K file: 62.5 tok/s vs 26.0 without MTP, 21/25 drafts accepted.
• Gate Huihui MTP on the F16 draft loader capability — setup only offers MTP for that family when the
  loader supports it.

Opt-in VRAM cap + quality mode

• Add opt-in total VRAM fraction cap through cache reserves — --vram-frac F sizes the expert cache so
  the engine's own use stays under F of the card; a floor is kept free and the plan is frozen pre-touch
  (post-touch readings only accept or refuse; no resize).
• Add opt-in uniform-GPU VRAM cap quality mode — --vram-cap-mode quality routes every expert miss
  through the grouped GPU kernel, no CPU fallback. Measured 20/20 token-identical outputs vs the
  uncapped run, 6,759 vs 9,397 MiB peak.
• Book late VRAM and refuse serve requests that cross the floor + cap booking: keep the named late allocations, leave graphs to the slack + refusal — books the chunked-GDN scratch and MTP prefill records
  in the frozen budget, re-checks the floor on the serve path and after cudaGraphInstantiate, and logs
  free memory at READY / first capture / first chunked launch / first checkpoint. Validated: free
  3,511–3,577 MiB against the 2,048 MiB floor, no refusals, prefix reuse intact.

qwen35moe prefill and routing

• Port opt-in chunked DeltaNet recurrence to qwen35moe — opt-in (STRATA_GDN_CHUNKED), parity-checked
  against the default recurrence.
• Use qwen35moe active expert count for router lookahead — width 8 instead of qwen4exp's 10.
• Port opt-in route-resident routing to 256-expert top-8 models — the 256/8 counterpart of route-resident: one warp per token in the swap kernel #1737.
• route-resident: one warp per token for the 512/10 and 256/8 swap — merges route-resident: one warp per token in the swap kernel #1737's warp-shuffle swap
  with the 256/8 implementation into one templated kernel (route_resident_impl<NE,K>). This commit can
  replace route-resident: one warp per token in the swap kernel #1737 if preferred: it contains its approach for 512/10 and adds 256/8.

Determinism, setup, SYCL

• Make expert-cache auto use one fixed 1 GiB haircut + Bucket the qwen35moe auto-cache haircut and check the reserve — fixes the auto-cache non-reproducibility on Windows/WDDM (2,974 vs 3,494 slots between
  identical runs) and refuses cleanly instead of stalling under low headroom.
• Add Huihui abliterated Qwen3.6 setup family — the family in setup.py with tests.
• Keep the shared Qwen36 setup test harness offline on Windows.
• qsa: mirror the dense-attention validate() fix into the SYCL twin — the CUDA fix in Qwen3.6-35B-A3B and Ornith-1.5-35B-A3B (qwen35moe): a second, smaller model for 8-12 GB cards #1402 did not touch
  the SYCL twin.
• SYCL generate: explicitly refuse the CUDA-only VRAM cap flags — fail closed instead of ignoring them.

Validation (RTX 3080 10 GB, Windows 11, CUDA 13.2, sm_86)

Measured on this exact branch (stacked on 48534f3):

• Build: VS 18 / Ninja / Release / sm_86, exit 0.
• Full ctest: 96 passed, 1 skipped (expert_multi_test), 1 failed (ple_parity, missing Flash-Next
  Q2_0 fixture, not present on this machine). The two cap tests are vram_cap_test (169 CPU checks)
  and vram_cap_quality_test (2,529 CPU checks); the other suites are the repo's own.
• Capped quality + MTP smoke on the Huihui Q4_K file: exit 0, output token ids identical to the validated
  line (64/64), --expert-cache 3200 capped to 1430 slots, free 3,513 MiB after the first capture.

Measured on the integrated line this work sits in (95f205d + the cap-floor fix, sha d58e8ded;
it also contains the five upstream PRs we adopted locally — #1700/#1660/#1659/#1696/#1720 — which this PR
deliberately excludes):

• Cap + quality: 20/20 token-identical vs uncapped, 6,759 vs 9,397 MiB peak.
• MTP: 62.5 vs 26.0 tok/s (2.4×), 21/25 accepted; server path 71.1 tok/s.
• Full stack (cap + quality + MTP + chunked + route-resident): 10/10 arms exit 0, whole-card peaks
  7,269–7,371 MiB under the 8,192 ceiling; server prefix reuse 1,005/1,012 with TTFT 1.89 s → 0.098 s;
  chunked kernel launches confirmed under Nsight Systems; HumanEval 39/40 at 2,048 and 39/40 at 4,096
  (no truncations; the single miss is HumanEval/132's wrong implementation).
• Determinism: byte-identical outputs across repeated runs at a fixed cache geometry; the bucketed
  haircut reduces but cannot eliminate sensitivity when the pre-touch reading crosses a bucket boundary, so
  a run can still open a different slot count. Cache size does not change the output tokens in quality mode
  (20/20 identical), but the slot count is not guaranteed bit-identical run to run. Clean refusal (no stall)
  when the reserve cannot be met.
• Backends: CUDA built and exercised; HIP and SYCL were not built (no toolchain on this machine). The
  SYCL changes are verified by inspection only, which does not meet AGENTS.md's build-every-touched-backend
  rule; an Intel/HIP machine should build them before this is merged.

What this leaves alone

• Defaults: cap, quality, chunked and route-resident are all opt-in. Two of the determinism commits do
  change the qwen35moe fast-mode auto-cache path: the bucketed haircut activates under WDDM for eligible
  native non-indexed models even without --vram-frac, and the router lookahead uses the model's active
  expert count (8 instead of 10). qwen4exp keeps its measured-shortfall loop; no other default path changes.
• qwen4exp: the route-resident change keeps the 512/10 behavior; the auto-cache fixes keep the
  measured-shortfall loop for qwen4exp.
• Not included: generate, mtp: near the context's end the spec window and its drafts stay inside it #1700 / prefill: let native-layout packs take STRATA_MMQ_RESIDENT_SORT_NE #1660 / prefill gemm: recover the FP16 shapes cuBLAS 12's default algorithm fails on (issue #1650) #1659 / Resident RAM mode on Windows: the start-up fill and the RAM copy read experts.bin unbuffered (ready 42 -> 10 s with 21 GiB free) #1696 / dequant: coalesced Q8_0 path (four threads per block, 16-byte stores), bit-identical, 7x faster on V100 #1720 (separate PRs) and route-resident: one warp per token in the swap kernel #1737 (superseded by the
  templated 512/10 + 256/8 commit here).

Follow-up fixes from an adversarial review (2026-10-10)

Three review findings were fixed on top of the stack:

• MTP F16: keep NaN NaN in the F16/BF16 conversion — the rounded addition carried NaN into infinity or a
  signed zero; NaN now keeps a nonzero BF16 mantissa.
• SYCL: refuse a malformed STRATA_VRAM_FRAC like the CUDA build does — 0, NaN, empty, trailing junk and
  underflow-to-zero were silently ignored; they now exit 2 as on CUDA.
• VRAM cap: check the floor at every chunk, at serve decode admission, before pipelined windows — the
  floor was checked only before the first prompt chunk; a long prefill, a fully reused prompt, and the
  --pipeline-windows decode loop could step past it. Prefill now has a per-chunk guard, serve re-checks
  at decode admission, all five pl_launch sites are guarded (the initial A, the speculative B, the
  promoted A's stage-1 launch, and both prompt-pipeline launches), and the first-use chunked-GDN scratch
  allocation checks the floor before taking its bytes.

Still open from that review (not fixed here): the stack does not meet the repo's one-change-per-PR rule,
and HIP/SYCL builds are not run.

Known limits

• One machine: Windows, RTX 3080, CUDA. No AMD/Intel or Linux numbers here.
• The F16 draft conversion is bit-checked and end-to-end on the Huihui GGUF; no other F16-draft model was
  available.
• Quality mode requires a native expert pack; on this machine it cannot run the 46.8 GiB Flash-Next arena
  because the whole arena cannot be pinned (a host limit, unrelated to these paths).
• The determinism fixes target the Windows/WDDM auto-cache path; Linux behavior was not re-measured here.

[aly8246]

The 256/8 generalization is exactly what this wanted. Two things that may save you work.

The win survives the new defaults. On 0.1.42 with STRATA_ROUTE_TAIL_SKIP at its new default of 7, this
kernel is worth +9.7% decode / -13.1% window (median of 5 paired ABBA rounds, 20 engine restarts, the
learned expert profile restored before every leg; the two builds differ by only these 7,680 bytes of kernel
code, and the greedy answers came out identical). The tail skip shortens the window, which makes the swap a
larger share of it, so the two do not cancel. For scale: +6.6% on 0.1.41, and 0.1.41 -> 0.1.42 by itself was
+11.4% decode.

A parity test, if the parameterized kernel does not have one yet. src/kernels/route_resident_parity.cu
in #1737 is 240 random cases against a scalar reference of the semantics - a serial walk with the same
comparisons, not a copy of the kernel it replaced - checking ids and the four counters bit for bit and the
weights within 8 ulp (worst seen: 6), with a sentinel that fails a no-swap case that touches the weights.

git cherry-pick 488eecba from that PR is the whole thing: the test file is new and the CMakeLists hunk is
8 lines. It keeps the authorship, which is nicer than a trailer if you want the provenance in the log - but
your commit message already names #1737, so take it as is, adapt it, or ignore it, all fine.

[atomicmilkshake]

Extracted the 256/8 route-resident generalization and parameterized parity test into a clean, standalone single-commit PR on top of \main: #2321.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.