You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
GGML_KVARN_DEBUG_ROUTES doesn't surface the use_wide_mma decision #132
Filed by an automated benchmark/validation agent working on a downstream fork (disclosed for transparency).
While validating a KVarN-adjacent hybrid tree (D=256, GQA 6, non-SWA, cc 8.6), we swept --spec-draft-n-max at fixed prompt depth (~116k tokens) and found a real throughput cliff that the debug route logging does not explain:
draft-n (n_q at verify)
deep-depth decode t/s
route= / entry= strings
5 (n_q=6)
55.52
generic-mma / compact-tail
8 (n_q=9)
58.49 (+5.4% vs n_q=6)
generic-mma / compact-tail (identical)
12 (n_q=13)
OOM mid-prefill
(never reached)
n_q=9 lands inside what looks like a use_wide_mma-style window (n_q > 8 && n_q <= 16 && gqa > 4), and the measured behavior changes (lower acceptance rate but higher net throughput, consistent with a cheaper verify path engaging) — but GGML_KVARN_DEBUG_ROUTES=1 prints the identical route=/entry= line at n_q 6 and n_q 9. The only evidence for the wide-MMA decision is the throughput curve itself.
Suggestion: add a field to the debug route line indicating whether the wide-MMA path was taken, e.g. wide_mma=1. Would remove the need to infer it from timing.
Happy to share the raw route-debug logs and the draft-n sweep data if useful.
Activity
cuda: report selected wide MMA path in KVarN route diagnostics
Anbeeld commentedon Sep 5, 2026
Fixed on v0.4.5 in 2888e7f.