Skip to content

ci(cuda): detect runner GPU architectures before building - #8347

Merged
mohanchen merged 1 commit into
deepmodeling:developfrom
Stardust0831:ci/cuda-runner-architecture
Nov 21, 2026
Merged

mohanchen merged 1 commit into
deepmodeling:developfrom
Stardust0831:ci/cuda-runner-architecture

Conversation

@Stardust0831

Copy link
Copy Markdown
Collaborator

The CUDA workflow can run on Tesla P100 (sm_60), but it currently builds only for sm_70. Detecting the runner's compute capabilities lets P100 build for 60 and V100 build for 70, avoiding CUDA error 209 without compiling unused architectures.

Linked Issue

No separate issue is needed: the failing develop CUDA job records a Tesla P100-PCIE-16GB, CUDA 12.2.140, CMAKE_CUDA_ARCHITECTURES=70, and no kernel image is available for execution on the device in snap_psibeta_kernel.cu.

What's changed?

Query nvidia-smi --query-gpu=compute_cap --format=csv,noheader inside the CI job, validate the output, and pass sorted, unique numeric architectures to CMake. Repeated devices produce one target; mixed allocations compile for each architecture present. Empty, invalid, or failed detection stops configuration.

Detection is limited to CI, where compilation and testing use the same GPU allocation. Keep the current build parallelism, shared compiler cache, and GPU test steps. Explicit numeric nvcc arguments distinguish CUDA objects while preserving C++ cache reuse across GPU types.

Unit Tests and/or Case Tests for my changes

  • Exercised the actual detection block with 12 GPU responses, including P100, V100, repeated/mixed devices, reordered devices, A100, two-digit major capabilities, empty output, malformed values, and query failure. All passed; the architecture list reaches CMake as one argument.

  • CUDA 12.2.140 / GCC 12.4.0: the detected V100 target produces only sm_70 and a CUDA probe runs successfully on Tesla V100-SXM2-16GB. Simulated P100 input 6.0 produces target 60 and only sm_60 machine code. P100 execution is not yet verified.

  • Built the existing ABACUS CUDA targets with detected architecture 70 using CUDA 12.9.86 / GCC 13.3.0. cuobjdump --list-elf confirms that the actual snap_psibeta_kernel.cu.o contains only sm_70. Build commands:

    cmake -S source -B build -DUSE_CUDA=ON -DENABLE_LIBXC=ON -DBUILD_TESTING=ON \
      -DENABLE_FLOAT_FFTW=ON -DENABLE_GOOGLEBENCH=OFF \
      "-DCMAKE_CUDA_ARCHITECTURES=$architectures" \
      -DFETCHCONTENT_SOURCE_DIR_GOOGLETEST="$PWD/googletest" \
      -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
    cmake --build build --parallel 32 --target \
      MODULE_LCAO_tddft_snap_psibeta_half_test MODULE_LCAO_radial_tables_gpu \
      MODULE_LCAO_tddft_rad_interp_cuda_test
  • With OMP_NUM_THREADS=1 on Tesla V100-SXM2-16GB, the following command passed all three targets: 8 snap tests, 5 radial-table tests, and 1 radial-interpolation CUDA test (14 tests total):

    ctest --test-dir build -V --timeout 1700 \
      -R '^(MODULE_LCAO_tddft_snap_psibeta_half_test|MODULE_LCAO_radial_tables_gpu|MODULE_LCAO_tddft_rad_interp_cuda_test)$'
  • With shared ccache 4.14.3 / CUDA 12.2.140, compiling the same source for 70 -> 60 -> 70 produced two misses and one direct hit; object inspection confirmed the requested architecture each time.

  • Bash, actionlint, focused ShellCheck, git diff --check, and repository governance checks passed. Whole-workflow ShellCheck reports the same four existing SC2162 informational findings as develop.

  • Still required: a complete CUDA workflow run on the upstream P100 runner. Full-workflow build timing and all GPU Autotest suites were not reproduced locally.

Governance Notes

No INPUT or core-module behavior changes; no documentation update or governance exception required. Existing CUDA tests exercise the resulting builds.

Query compute capabilities in the CI allocation and pass unique numeric
architectures to CMake. Keep single-architecture builds on homogeneous
runners and preserve the existing compiler cache and test steps.
@mohanchen mohanchen added Bugs Bugs that only solvable with sufficient knowledge of DFT Refactor Refactor ABACUS codes labels Nov 21, 2026
@mohanchen
mohanchen merged commit 36a3ebf into deepmodeling:develop Nov 21, 2026
20 checks passed
@mohanchen mohanchen added the GPU & DCU & HPC GPU and DCU and HPC related any issues label Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bugs Bugs that only solvable with sufficient knowledge of DFT GPU & DCU & HPC GPU and DCU and HPC related any issues Refactor Refactor ABACUS codes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants