Repository navigation
Feature(hsolver): add LOBPCG for plane-wave calculations - #8358
Cstandardlib wants to merge 12 commits into
Conversation
Implement a generalized LOBPCG iteration for CPU and GPU with hard/soft locking, optional extra bands, and reusable Rayleigh-Ritz workspaces. Integrate the solver into HSolverPW and INPUT metadata, with focused unit tests and CPU LAPACK/ScaLAPACK SCF cases.
ABACUS-PR-BOT
left a comment
There was a problem hiding this comment.
Static review found issues in iteration-budget handling and ELPA edge cases. No builds, numerical/MPI/GPU runs or CI diagnostics were performed; the reported validation remains author-reported.
Non-blocking: Please add the tested commit, build options, exact test/SCF commands and stable case/input identifiers behind the reported results, distinguishing CPU, CUDA and ROCm coverage. Relevant logs would make the accuracy, performance and memory claims reproducible.
Use a small float buffer and remove near-full-space and large dense stress cases. Check convergence, reference eigenvalues, orthogonality and padding without imposing a separate residual stopping criterion.
Pass pw_diag_nmax through the common PW setup for all calculation modes and solvers. Reject nonpositive budgets in INPUT validation. Continue electronic SCF after LOBPCG budget exhaustion in relax, cell-relax and MD, while retaining an error for unconverged NSCF solves. Cover budget propagation across methods/modes, caller exit behavior, and positive INPUT values with focused functional tests. Validation: MODULE_HSOLVER_pw and MODULE_IO_lobpcg_input pass (10 unit tests). Staged governance check has no errors. Documentation: the INPUT domain metadata supplies the generated positive value constraint; no separate generated documentation needs editing.
ABACUS-PR-BOT
left a comment
There was a problem hiding this comment.
Incremental static review of 42c203bd..14e8c86f found two gaps in numerical test coverage. No runtime checks were performed.
Reject lobpcg with diag_subspace=1 during INPUT validation and report supported values 0/2 to direct callers. Remove the LOBPCG ELPA adapter and its test dependencies; defer ELPA support to a follow-up PR. Keep public ELPA and Davidson implementations unchanged. Update INPUT metadata, CLI help and generated documentation descriptions to state the supported combinations, including CUDA cuSOLVER for option 0. Validation: 8 CPU and 4 GPU CTest targets pass. CLI --check-input accepts LOBPCG 0/2, rejects LOBPCG 1, and accepts Dav_Subspace 0/1/2. Generated input-main and parameter help were checked. Governance has no errors. Documentation is generated from the updated Input_Item metadata; generated YAML and Markdown are not committed.
Use a small diagonal generalized problem with distinct eigenvalues to check returned vector columns against their analytic eigenvectors up to phase. Exercise the iterative path in float and double with hard and soft locking, and include the check in the GPU MPI2 test selection. Validation: four CPU/GPU single-process and MPI2 CTest targets pass. Production code and existing accuracy assertions are unchanged. No user documentation update is needed for this test-only change.
Set pw_diag_nmax=30 explicitly for MemSaver, Wannier90, and HSE NSCF fixtures to preserve the effective budget used to generate their references. NSCF now honors INPUT budgets, so leaving these fixtures unspecified would select 50 and change CG restart and near-degenerate eigenvector paths. Keep the production parameter fix, reference outputs, and comparison thresholds unchanged. Validation: existing Autotest harness passes all three cases on CPU (4 MPI ranks) and MemSaver/Wannier90 on GPU (1 MPI rank), with one OpenMP thread. Staged governance and diff whitespace checks pass.
ABACUS-PR-BOT
left a comment
There was a problem hiding this comment.
Source review at ee40021f confirms the revised budget handling and removal of the LOBPCG ELPA path address the corresponding earlier findings. The new diagonal fixture provides an independent eigenpair oracle. One new non-blocking test-fixture suggestion is inline, with source-status updates in the original threads. This was static review only; no builds or tests were run, and reported runtime/CI results remain unverified.
| SOURCES test_diago_lobpcg_float.cpp ${LOBPCG_TEST_SOURCES} | ||
| ../../source_basis/module_pw/test/test_tool.cpp) | ||
| set_tests_properties(MODULE_HSOLVER_lobpcg MODULE_HSOLVER_lobpcg_float PROPERTIES | ||
| TIMEOUT 180 ENVIRONMENT "OMP_NUM_THREADS=1;OPENBLAS_NUM_THREADS=1;MKL_NUM_THREADS=1") |
There was a problem hiding this comment.
this line may be too complicated, try to simplify the cmake
Resolve ESolver call-site conflicts with upstream formatting while preserving the LOBPCG extra-band and locking constructor arguments. Keep all other upstream changes and the existing LOBPCG implementation. Validation: CPU/LibXC and CUDA PW executables build. Existing Autotest checks pass for six CPU cases (LOBPCG 0/2, three NSCF regressions and SDFT) and three GPU cases (LOBPCG, MemSaver and Wannier90). Diff whitespace and conflict checks pass; staged governance has no errors. No INPUT or user-facing behavior changes are introduced by the resolution.
Conjugate stored Hermitian matrix rows when accumulating distributed output columns so the test operator applies H rather than H transpose. Add a two-dimensional complex matrix-vector check with exact analytic outputs, independent of the iterative eigensolver convergence criteria. The new check fails with the previous operator. After the fix, both MODULE_HSOLVER_lobpcg_float and its MPI2 variant pass (14 GTests total). Existing convergence thresholds and assertions are unchanged. No user documentation update is needed for this test-only correction.
Set thread limits once for the registered LOBPCG tests, reuse the common MPI test helper source list, and select exit-check rank counts once. Keep backend-specific registration and properties explicit. CTest JSON comparison confirms unchanged commands and properties for all 24 CPU and 31 CUDA-build HSolver test registrations, including the existing timeouts, MPI flags, filters and device skip behavior. Validation: six CPU and four GPU LOBPCG CTest targets pass, covering single-process, MPI2 and fatal-exit checks. Staged governance passes. No user documentation change is needed for this test-build refactor.
Let LOBPCG tests inherit the runtime environment and run the complete GPU MPI2 suite without a test-name allowlist. Register exit checks next to each device target through a small local helper, and build their MPI launcher once before running the scenarios. Use explicit test target names, merge adjacent GPU configuration blocks, consolidate HSolverPW test sources, and remove an unrelated IO directory comment. Preserve existing timeout, MPI flag and GPU skip behavior. Validation: CPU 8/8 and GPU 4/4 CTest targets passed after the registration cleanup. After removing the GPU MPI2 filter, the full 27-test GTest suite passed with OMP_NUM_THREADS=2 and no OpenBLAS/MKL thread override. CTest metadata comparisons confirm the intended environment/filter changes and preserve the other test commands and properties. Governance and diff whitespace checks pass. No user documentation change is needed for this test-configuration cleanup.
ABACUS-PR-BOT
left a comment
There was a problem hiding this comment.
The update retains the corrected float operator fixture and expands GPU MPI selection. One test-environment suggestion is inline. This was static source review only; no builds or tests were run, and reported runtime/CI results remain unverified.
| "-DMPI_PREFLAGS=${MPIEXEC_PREFLAGS}" "-DMPI_POSTFLAGS=${MPIEXEC_POSTFLAGS}" | ||
| -P "${CMAKE_CURRENT_SOURCE_DIR}/test_diago_lobpcg_exit.cmake") | ||
| set_tests_properties(${exit_name} PROPERTIES PROCESSORS ${ranks} TIMEOUT 240 | ||
| SKIP_REGULAR_EXPRESSION "SKIP: no usable GPU for LOBPCG exit checks") |
There was a problem hiding this comment.
Non-blocking: Please preserve the shared thread limits for all LOBPCG test registrations. The deleted ENVIRONMENT block was the only setting for OMP_NUM_THREADS, OPENBLAS_NUM_THREADS and MKL_NUM_THREADS, and AddTest provides no replacement; callers with multithreaded defaults can therefore use extra CPU resources and face avoidable timeout risk. Restore the common assignment after the CPU/GPU, MPI and exit tests are registered.
Summary
Add the locally optimal block preconditioned conjugate gradient method (LOBPCG, Toward the Optimal Preconditioned Eigensolver: Locally Optimal Block Preconditioned Conjugate Gradient Method, Knyazev 2001) as
ks_solver=lobpcgfor plane-wave calculations. It supports standard and generalized eigenproblems for norm-conserving (NC) and ultrasoft (US) pseudopotentials, on CPUs and GPUs.Select it with
basis_type=pwandks_solver=lobpcg. Different locking technique (lobpcg_locking) for eigenvectors and a number of extra eigenvectors (lobpcg_extra_bands) to form the searching subspace provide optional tuning and potential acceleration for slowly converging bands. The defaults are hard locking and zero extra bands, which demonstrates the best performance on most systems. See INPUT documents for details of the solver and its options.Correctness
Converged energies agree closely with CG, DAV, and Dav_Subspace across the plane-wave benchmark suite. The table reports the absolute energy difference from CPU and GPU for each system (
scf_thr=1e-7, eV/atom).Representative NCPP and USPP calculations also agree in forces and stresses as SCF convergence is tightened. Results below compare LOBPCG with Dav_Subspace at
scf_thr=1e-12; force and stress entries are maximum component differences.These systems derive from
tests/01_PW.Performance
On this suite, LOBPCG is competitive with Dav_Subspace on CPU and substantially faster overall than CG and DAV. Dav_Subspace remains faster overall on GPU.
CPU: 28 MPI ranks with
diag_subspace=2(ScaLAPACK as inner eigensolver); GPU: one rank on one RTX 3090. LOBPCG uses hard locking, zero extra bands. CPU plots cover ten systems; GPU plots cover the nine systems where 216Si runs out of device memory.CPU
Wall time:

Relative:

GPU
The GPU version has not been optimized so far. Dav_Subspace remains the best choice on GPU.

Optional locking and extra bands
Soft locking accelerates some calculations, for example GaAs and BaTiO₃ as demonstrated below. A small buffer (extra bands, which usually takes fewer steps to converge but is more expensive per iteration) also reduced GPU time for 27Fe in the tested run. Benefits depend on the system and device; hard locking with zero extra bands remains the default and optimum configurations for most systems.
Memory
LOBPCG reduces memory use relative to Dav_Subspace on larger systems, making it useful when memory capacity matters. Examples use sampled peak CPU host memory (PSS, diag_subspace=2) and GPU device memory.
Verification and integration
scf_lobpcgandscf_lobpcg_scaintegration cases for complete SCF calculations with LAPACK and ScaLAPACK.