Skip to content

prefill: ~7% pp512 regression on Arc B580 since #30160 (embeddings get_rows reorder), constant per-invocation cost #30939

Open

Description

@dosermen

Setup: Intel Arc B580 (Xe2), driver 32.0.101.9150, Windows 11, MinGW Release build (static, Vulkan backend), llama-bench -t 14 -ub 512 -p {512,4096,16384} -n 128 -fa auto, gemma-4-E4B Q4_0 (dense, standard token path, no raw embeddings, no multimodal inputs), warm card.

Bisecting the prefill regression over 448147d42..b11756 (279 commits, stock builds, no local patches) isolates two independent commits: first bad is 5e4878e97 (#30145, tracked separately in #30381), second bad is 8e2d31e0e (#30160, "graph : reorder get_rows for embeddings").

pp512, stock builds:

build pp512
448147d42 (07.10) 3356 t/s
b11756 3102 t/s (-7.6%)

Bisect details:

Second confirmation on a local integration branch, interleaved A/B (4 pairs, binaries alternated within a single session; identical-binary control: mean -0.4%, single run +/- 2%; my branch carries a local FA capability-gate relaxation for B580, see the provenance note in #30018 - deltas are internally consistent, absolute values on stock builds differ):

tree pp512
pre-rebase (predates #30160) 4294 t/s
b11756 + local fixes + local revert of #30145 4019 t/s (-6.9%, 4/4 pairs)

Re-enabling #30145 on top of the same tree costs another -3.9% (3995 -> 3838 t/s), consistent with #30381.

Characteristics of the #30160 cost:

  • Prefill-only and confined to short prompts: pp512 -6.9% (the only statistically solid point: interleaved, +8.17 ms per invocation); pp4096 -1.2% and pp16384 ~0% are within single-shot noise (+/- 2%). Profile is consistent with a one-time per-invocation overhead (~8 ms), not a per-token slowdown.
  • Same delta with -fa off and -fa auto; per-op GPU time sums are unchanged across the boundary, so the cost is host-side (graph build / first-batch sync), not shader time.
  • decode (tg128) is not affected by this cost; across the whole commit range decode actually improved +5.7%, not attributable to this commit.

#30160 is a correctness fix (mixed/raw-embedding branches, gemma4 PLE, etc.), so I am not proposing a revert. Question: is the extra one-time cost on the plain token-only path expected? For a dense model with only token embeddings the reordered split-0 construction appears to add a host-side sync that did not exist before; if the new ordering could be limited to graphs that actually need it (mixed batch / raw embeddings), the standard path would keep the old timing.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions