Skip to content

prefill gemm: recover the FP16 shapes cuBLAS 12's default algorithm fails on (issue #1650) - #1659

Closed
W1nge wants to merge 1 commit into
Niko1221:mainfrom
W1nge:codex/cublas-f16-algo-fallback
Closed

W1nge wants to merge 1 commit into
Niko1221:mainfrom
W1nge:codex/cublas-f16-algo-fallback

Conversation

@W1nge

@W1nge W1nge commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Issue #1650: with more than one CUDA device initialized in the process, cuBLAS 12 answers the default
algorithm of a few FP16 GEMM shapes (CUDA_R_16F operands, CUBLAS_COMPUTE_32F) with
CUBLAS_STATUS_INTERNAL_ERROR (status 14), and the request aborts with
prefill gemm: cublasGemmEx f16: cuBLAS status 14.

Only initializing the other device's context is enough to set the state that does it: on this host,
cudaSetDevice(1); cudaFree(0); on the P100 makes the default-algorithm FP16 GEMM fail on the 2080 Ti
before any work has run there. The failing shapes depend on K as well as N (K = 10240: N a multiple
of 8 is safe; K = 2560: N = 8..32 fails too), so a "small N" guard is not enough.

The fixed algorithms (ALGO2 and up) run through that state, but they are markedly slower for the wide
shapes. So a Gemm moves only the shapes that failed: it remembers their (N, K) ((K << 32) | N, 32
slots) and runs them on CUBLAS_GEMM_ALGO2. The failing call is harmless and is paid once per shape
per process.

Measured on one Windows host (CUDA 12.4.131, RTX 2080 Ti sm_75 + Tesla P100 sm_60, driver 537.13), the
3-request fixture with STRATA_BF16_TC=1 (the abort is the stock engine, not a number):

FP16 algorithm 2K prompt 5.1K prompt
stock (CUBLAS_GEMM_DEFAULT) aborts aborts
every shape on ALGO2 7113 ms 11515 ms
this change 6261 ms 7499 ms

For reference, the same fixture with the tensor cores off is 6333 ms / 7899 ms: this change keeps the
STRATA_BF16_TC win and removes the abort.

Tested: the stock engine aborts on --short-read 0 (40-token prompt, STRATA_BF16_TC=1,
cublasGemmEx f16: cuBLAS status 14) and this change answers it ([3], valid, 3.1 s); the 2K / 5.1K /
warm fixture and the daily lend/source regression all pass with the same answers.

…he default one fails once a second device's context exists (issue Niko1221#1650)
@Niko1221

Copy link
Copy Markdown
Owner

Ported with your authorship kept. Builds on CUDA 12.9 and 13.4, identity gate unchanged on a 3060. We could not trigger the cuBLAS 12 state on our same-arch pair, so thanks for the 2080 Ti + P100 data.

@Niko1221 Niko1221 closed this Oct 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants