Repository navigation
Conversation
…he default one fails once a second device's context exists (issue Niko1221#1650)
Owner
|
Ported with your authorship kept. Builds on CUDA 12.9 and 13.4, identity gate unchanged on a 3060. We could not trigger the cuBLAS 12 state on our same-arch pair, so thanks for the 2080 Ti + P100 data. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #1650: with more than one CUDA device initialized in the process, cuBLAS 12 answers the default
algorithm of a few FP16 GEMM shapes (
CUDA_R_16Foperands,CUBLAS_COMPUTE_32F) withCUBLAS_STATUS_INTERNAL_ERROR(status 14), and the request aborts withprefill gemm: cublasGemmEx f16: cuBLAS status 14.Only initializing the other device's context is enough to set the state that does it: on this host,
cudaSetDevice(1); cudaFree(0);on the P100 makes the default-algorithm FP16 GEMM fail on the 2080 Tibefore any work has run there. The failing shapes depend on K as well as N (K = 10240: N a multiple
of 8 is safe; K = 2560: N = 8..32 fails too), so a "small N" guard is not enough.
The fixed algorithms (ALGO2 and up) run through that state, but they are markedly slower for the wide
shapes. So a
Gemmmoves only the shapes that failed: it remembers their(N, K)((K << 32) | N, 32slots) and runs them on
CUBLAS_GEMM_ALGO2. The failing call is harmless and is paid once per shapeper process.
Measured on one Windows host (CUDA 12.4.131, RTX 2080 Ti sm_75 + Tesla P100 sm_60, driver 537.13), the
3-request fixture with
STRATA_BF16_TC=1(the abort is the stock engine, not a number):CUBLAS_GEMM_DEFAULT)For reference, the same fixture with the tensor cores off is 6333 ms / 7899 ms: this change keeps the
STRATA_BF16_TCwin and removes the abort.Tested: the stock engine aborts on
--short-read 0(40-token prompt,STRATA_BF16_TC=1,cublasGemmEx f16: cuBLAS status 14) and this change answers it ([3], valid, 3.1 s); the 2K / 5.1K /warm fixture and the daily lend/source regression all pass with the same answers.