[aslater3]

Title

sycl/A770: faster hyper-connection read on Arc A-series (gr_bench 88 / 123 / 151 -> 62 / 83 / 100 us at 1 / 4 / 6 tokens)

Summary

fused_gr_read_multi on Alchemist, the gr_more_groups path. This is the decode window's per-layer hc read, run twice per layer, and it sits on the stepped window's critical path. There are five changes, each with an env switch and each the default on Alchemist only. Every other device runs the same kernels as before.

Three are bit-identical (gr_bench checksums unchanged):

1. The down reduce is folded into the sliced down kernel: each row block's last slice to finish sums the 80 slice partials, in the same order (STRATA_GR_FUSED_REDUCE). That is one launch fewer.
2. The up kernel runs at 4 columns per block (640 blocks), up from 8 (STRATA_GR_UP_MORE_COLS).
3. The up kernel reads the lo values the fused reduce computed once, instead of every block recomputing them (STRATA_GR_LO_READY).

Two change the fp32 summation order. The output sums agree to about 1e-7 relative, and the output was quality-checked:

4. The norm is folded into the down kernel (STRATA_GR_NORM_FOLD). The slices stage the pending-updated residual × w_norm unscaled, together with its sums of squares. The reduce then applies each stream's rs to that stream's partial dots. This removes the norm launch and the xn write and re-read.
5. The up projection runs one row per lane (STRATA_GR_UP_V5). The old kernel re-read all of lo from local memory for every row and every token, about 160 KB per block, and was local-memory bound on the A770. Now all lanes read the same lo value at once, as a broadcast.

STRATA_GR_NORM_FOLD=0 STRATA_GR_UP_V5=0 gives back the old bits exactly.

What changed

• sycl/src/kernels/cuda/fused_gr.dp.cpp:
  • gr_down_sliced_kernel gains the FUSED / NF epilogues.
  • New gr_up_v5_kernel.
  • launch_gr_down_more / launch_gr_up_more choose between them.
  • New switches: gr_fused_reduce, gr_norm_fold, gr_up_more_cols, gr_lo_ready, gr_up_v5.
  • New scratch: slice counters and slice sums of squares.

Extra Notes

Arc A770 16 GB, xe, compute runtime 26.05, oneAPI 2026.1. gr_bench, in µs:

tokens	before	1-3 (bitwise)	+ 4	+ 5
1	88.4	74.6	62.2	63.8
4	123.1	104.9	90.3	83.3
6	150.8	133.2	115.4	100.5

Decode (Coder IQ1_M, --resident-experts, spec 4):

• 1-3: 33.64 / 33.41 → 34.80 / 34.84 tok/s with the expert large-GRF PR, ids identical.
• 4-5: GPU wait per round 39.2-39.9 → 38.1-39.2 ms at 8K / 32K / 2K. Decode is within run-to-run noise (about +1%). The generated text is equivalent, and the 2K ids are identical.
• Rejected along the way: 8 lanes per row for the up kernel (slower), an all-80-in-flight reduce (spills), and 2 / 16 / 32 columns.

Not tested: A750 / A580, i915, Windows.

🤖 Generated with Claude Code

[aslater3]

Added 607f211c: the norm-folded down kernel now uses 256-column slices on Alchemist (STRATA_GR_DOWN_COLS, with 128 / 256 / 512 selectable). Each row block's last slice then reduces 40 partials instead of 80, and every lane's weights are loaded before the staging.

gr_bench gains GR_BENCH_LAYERS=N, which runs 48 layers' weights from VRAM (the timings are the same as with a single, L2-resident copy).

tokens	128 columns	256 columns
2	68.8 us	62.9 us
4	83.4 us	78.4 us
6	100.5 us	95.3 us

• Engine: at 8K the GPU wait per round went 37.2-37.7 -> 35.9-36.1 ms over 3 same-day pairs.
• Rejected: 512 columns spill and are 2x slower, and so is a 2-rows-per-lane variant.
• Numerics: this is another fp32 order than the 128-column kernel, as the norm fold already was. The output sums agree to ~1e-7.

🤖 Generated with Claude Code

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.