You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
prefill: ~7% pp512 regression on Arc B580 since #30160 (embeddings get_rows reorder), constant per-invocation cost #30939
Setup: Intel Arc B580 (Xe2), driver 32.0.101.9150, Windows 11, MinGW Release build (static, Vulkan backend), llama-bench -t 14 -ub 512 -p {512,4096,16384} -n 128 -fa auto, gemma-4-E4B Q4_0 (dense, standard token path, no raw embeddings, no multimodal inputs), warm card.
Bisecting the prefill regression over 448147d42..b11756 (279 commits, stock builds, no local patches) isolates two independent commits: first bad is 5e4878e97 (#30145, tracked separately in #30381), second bad is 8e2d31e0e (#30160, "graph : reorder get_rows for embeddings").
phase 2 (with 5e4878e97 locally reverted at every step, 230 commits): first bad = 8e2d31e0e (graph : reorder get_rows for embeddings #30160). A plain git revert of it no longer applies on master (later commits build on top of it), so each bisect step was revert -> build -> bench -> reset.
Second confirmation on a local integration branch, interleaved A/B (4 pairs, binaries alternated within a single session; identical-binary control: mean -0.4%, single run +/- 2%; my branch carries a local FA capability-gate relaxation for B580, see the provenance note in #30018 - deltas are internally consistent, absolute values on stock builds differ):
Prefill-only and confined to short prompts: pp512 -6.9% (the only statistically solid point: interleaved, +8.17 ms per invocation); pp4096 -1.2% and pp16384 ~0% are within single-shot noise (+/- 2%). Profile is consistent with a one-time per-invocation overhead (~8 ms), not a per-token slowdown.
Same delta with -fa off and -fa auto; per-op GPU time sums are unchanged across the boundary, so the cost is host-side (graph build / first-batch sync), not shader time.
decode (tg128) is not affected by this cost; across the whole commit range decode actually improved +5.7%, not attributable to this commit.
#30160 is a correctness fix (mixed/raw-embedding branches, gemma4 PLE, etc.), so I am not proposing a revert. Question: is the extra one-time cost on the plain token-only path expected? For a dense model with only token embeddings the reordered split-0 construction appears to add a host-side sync that did not exist before; if the new ordering could be limited to graphs that actually need it (mixed batch / raw embeddings), the standard path would keep the old timing.
Activity