Skip to content

cuda : fold cuBLAS conversion into tensor affine activation - #136

Draft
GenerelSchwerz wants to merge 10 commits into
reference/upstream-kernels-20261001from
kernel/cublas-affine-conversion-20261006
Draft

GenerelSchwerz wants to merge 10 commits into
reference/upstream-kernels-20261001from
kernel/cublas-affine-conversion-20261006

Conversation

@GenerelSchwerz

@GenerelSchwerz GenerelSchwerz commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

Fold tensor MUL -> ADD -> SIGMOID/SiLU -> optional SCALE into the existing typed cuBLAS output conversion. This covers a complete F32 result or a canonical column window repeated across every row and batch, retaining the full raw matrix and all live stages.

The original vendor/precision selection, strict input ownership checks, allocation dependencies and earlier HC/GLU/topk priority stay in place. Other windows remain available through the raw result. Unsupported layouts, intervening compute, direct-F32 output and native/MMQ/MMVQ paths keep their existing execution.

Component measurements

Six alternating pairs of 2000 graph replays on RTX 5070 Ti. These are separate comparisons; percentages are not additive.

Change and own control Cases Less component time
Whole result vs PR135 6c02602660e9 Four F16 cases 5.085%, 6.512%, 1.599%, 1.573%; 6/6 each
Column window vs previous PR136 3df94da2df30 Pre4, offset4, odd padded, large, T3 vendor 4.644%, 4.755%, 0.799%, 1.846%, 3.844%; 6/6 each

BF16/intervening/whole/native control paths are flat in the window comparison. Whole/native measure 0.019% / 0.127% more time, each faster in 3/6 pairs. No model-serving, MTP acceptance, private-executor or older-GPU runtime speedup is claimed.

Validation

  • Latest window extension: 19 matched cases, 38 profiles and 1952 CPU stage comparisons across four updated datasets; complete raw/window/live-stage/control/reference/upload bytes and remaining launch/resource maps checked. Original 5e-4 bound; worst stage NMSE 6.4911e-6.
  • Full-width/tail windows, H1/H7/H32, forty coefficient views, computed inputs/later raw readers, changed parameters/graph recapture, nonfinite values and original GLU priority covered. Computed-weight/later-reader arena grows 10 KiB; the other 18 arenas equal control.
  • Existing backend suite: 63/63 per arm at original 1e-7, including 12 new scalar/batched window cases. Run test-backend-ops -b CUDA0 -o DSV4_HC_PRE,MUL_MAT_UNARY with each qualified library directory.
  • Eight clean sanitizer jobs: five memcheck/leak checks, racecheck, and matched synccheck with 1024 tracking slots; 504 additional CPU stages and exact baseline bytes. No new Xid.
  • Eight fresh SM75/80/89/90 host/unary compiles, compile only. All 129 prior kernels/resources/9416 encoded instruction rows unchanged on SM120a; one new window kernel, 22 registers, no stack/local memory, +9336 bytes CUDA library.

Earlier whole-result qualification remains recorded separately: 21 matched cases, 51/51 stock tests per arm, eight sanitizer jobs and older compile checks. The extension preserves that converter and validates its unchanged execution again. Raw scripts, patches, hashes and records are under /home/gencoolpc/moe-cache-tests/results/upstream-kernel-series-20261001/HYPERCONNECTION/PACKED-CUBLAS-AFFINE-ASSESSMENT; earlier whole-result records are in sibling CUBLAS-AFFINE-CONVERSION.

This PR depends on PR135 and its listed HC/scalar prerequisites, with official reference/upstream-kernels-20261001 as review base. It folds one eligible window at conversion; it does not fold arbitrary later branches or change model-specific upstream operator limits.

AI assistance: Codex implemented and tested the changes; bounded independent GPT-6.1 Sol High review checked source and evidence. The owner handles upstream submission.

Carry the reviewed kernel sources from PR125, PR129 and PR132 as the
matching control for the separate tensor-affine HC follow-up.

Assisted-by: Codex
Preserve gate intermediates, original rounding and live HC outputs.
Retain strict input alias checks and standard allocation dependencies.

Assisted-by: Codex
Use the exact tested control sources from published PR130, PR129 and PR132.

Assisted-by: Codex
Reuse the allocated affine match to find the planned HC_POST and retain prepared emission priority. Reject expanded normalization cuts whose terminal is not written by this kernel.

Assisted-by: Codex
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant