[Benchmark] 2x Radeon AI PRO R9700 (gfx1201, TRX40 PCIe 4.0 x16): DSpark auto-disable policy misfires on 2-GPU TP=2 systems — --dspark on gives +35/+44/+52% decode #4194

Setup

• GPU: 2x Radeon AI PRO R9700 (gfx1201, 32 GB each)
• Platform: AMD Threadripper 3970X (32C) on Gigabyte Designare TRX40 — PCIe 4.0 x16 per slot, negotiated (sysfs on each card's upstream bridge port: current_link_speed=16.0 GT/s, current_link_width=16; the bridge/GPU silicon advertises up to 32.0 GT/s i.e. Gen5-capable hardware, the internal switch→die links inside each card run at Gen5 x16; host link is platform-limited to Gen4 x16 as designed)
• RAM: 128 GB DDR4 quad-channel (8 DIMMs)
• Model storage: Crucial P3 1 TB (DRAM-less NVMe) holding the pack (92% full)
• OS: Ubuntu 24.04.4, kernel 7.0.0-38, ROCm 7.2.4
• Engine: strata-dsv41 0.1.49.2 build d2d198a49, Linux source build; weights: DeepSeek-V4.1-Flash GSQ-RCO IST 2.00bpw (IQ2_XXS mixed set, 256.8 GB pack, zero-copy recipe from docs/DEEPSEEK41.md)
• TP=2, 100% native Strata memory management — no manual residency/pinning/placement tuning anywhere in this report

Protocol

One-shot engine launch per run, fixed ids file (same prompt across runs), 128 tokens generated, temperature 0. Per context length: 1 warm-up + 3 measurement runs. Decode/prefill rates taken from the engine's own decode:/prefill: output lines. 12+12 runs, all rc=0. The OFF arm uses the default --dspark auto (which the engine resolved to "disabled" here — see counters below); the ON arm uses --dspark on. Weights (256.8 GB) stay on NVMe; the "pack" is the zero-copy metadata pack (~9 MB) pointing at them in place, per docs/DEEPSEEK41.md.

Results (median, spread in parentheses)

VRAM is identical in both arms: max 31.61/31.63 GiB per card (native placement, never OOM). Engine VmHWM 44.7 → 44.3 GiB. Prefill is unaffected by the drafter in principle; the 1K delta reflects warm-up, the 8K −4% is within run-to-run noise/cache state.

DSpark was silently off with --dspark auto — native counters

With --dspark auto (default), all 12 runs show:

dspark: 127 rounds, 0 drafts offered, 0 accepted (1.01 tokens a round)

plus the engine's own message:

ds4: DSpark not loaded: the routed experts (132 GB) are far more than the GPU holds (67 GB), where its verify rounds read 3.3x a token's bytes and gain nothing (--dspark on loads it anyway)

With --dspark on, same protocol:

dspark: 44 rounds, 128 drafts offered, 83 accepted (2.91 tokens a round); drafting 5.76 ms a round, verifying 3021.0 ms in all
dspark per position (kept / reached): 1: 33/40 82%  2: 17/30 57%  3: 15/17 88%  4: 10/12 83%  5: 8/9 89%

Across lengths: 43–47 rounds per 128 tokens, 67–71% of draft positions accepted, 2.72–2.98 tokens per forward pass of the main model.

The problem

The auto rule (ds4_model.cu, ~L1998-2005) compares routed-expert bytes against free VRAM aggregated across all TP devices (per-device free0 minus the configured reserve): 132.1 GB of routed experts vs 67 GB of aggregate free VRAM, threshold 1.5x → 132.1 > 100.5 → drafter disabled. On this 2-GPU TP=2 box the empirical outcome is the opposite of the "gain nothing" prediction: forcing the drafter is strictly better at every length, with the gain growing with context (+52% at 8K), and peak VRAM unchanged (the native memory manager absorbs the 7.98 GB Q8_0 drafter without increasing peak VRAM usage or causing OOM in our measurements; we deliberately do not claim identical expert residency, which identical peaks do not prove).

Note this also differs from the docs framing that DSpark only pays off at 4xR9700: on this 2xR9700 box it already pays off heavily — it just never gets a chance because the heuristic disables it.

Suggestion

The static bytes-ratio seems to be the wrong axis, not just its scale: with the offload store active, the verify rounds read the routed experts through the same streaming path that already services ordinary decode steps, so "expert bytes vs free VRAM" overestimates the verify cost. Concretely:

• make the ratio configurable (e.g. --dspark-auto-threshold N); on this box the effective ratio is 1.97 (132.1/67) and the outcome at that ratio was +52%, so the working range for offloaded-weights setups is above the hard-coded 1.5,
• or gate auto on the observed economics: run a short verify round and disable the drafter only if measured acceptance/verify cost says so,
• at minimum: when auto disables the drafter, print that --dspark on is worth measuring on this class of system (offloaded experts, TP>=2), since here auto is a silent 30-50% performance cliff.

Also worth reconciling with #4123 (Maya streams the DSpark windows) — if the verify-cost model changes, this heuristic likely needs the same revisit.

Happy to provide raw bench.tsv, engine logs and sampler CSVs (VRAM/RAM/page-faults, 120 samples per run) if useful.