Skip to content

Feature(hsolver): add LOBPCG for plane-wave calculations - #8358

Open
Cstandardlib wants to merge 12 commits into
deepmodeling:developfrom
Cstandardlib:feature/lobpcg-pw
Open

Cstandardlib wants to merge 12 commits into
deepmodeling:developfrom
Cstandardlib:feature/lobpcg-pw

Conversation

@Cstandardlib

@Cstandardlib Cstandardlib commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add the locally optimal block preconditioned conjugate gradient method (LOBPCG, Toward the Optimal Preconditioned Eigensolver: Locally Optimal Block Preconditioned Conjugate Gradient Method, Knyazev 2001) as ks_solver=lobpcg for plane-wave calculations. It supports standard and generalized eigenproblems for norm-conserving (NC) and ultrasoft (US) pseudopotentials, on CPUs and GPUs.

Select it with basis_type=pw and ks_solver=lobpcg. Different locking technique (lobpcg_locking) for eigenvectors and a number of extra eigenvectors (lobpcg_extra_bands ) to form the searching subspace provide optional tuning and potential acceleration for slowly converging bands. The defaults are hard locking and zero extra bands, which demonstrates the best performance on most systems. See INPUT documents for details of the solver and its options.

Correctness

Converged energies agree closely with CG, DAV, and Dav_Subspace across the plane-wave benchmark suite. The table reports the absolute energy difference from CPU and GPU for each system (scf_thr=1e-7, eV/atom).

System vs CG vs DAV vs Dav_Subspace
4GaAs 6.48e-08 4.29e-08 7.54e-09
32H2O 2.23e-09 5.51e-10 2.04e-09
C2H6O 2.87e-08 4.26e-08 1.43e-08
12Pt111 6.38e-09 2.90e-08 5.34e-09
4MoS2 1.72e-08 2.36e-08 1.44e-08
16Na 2.51e-08 7.25e-09 1.23e-09
3BaTiO3 7.04e-08 4.25e-09 1.73e-09
LiNMC 2.21e-09 3.21e-09 2.40e-09
27Fe 1.14e-08 1.59e-08 1.56e-08
216Si 1.90e-10 2.54e-11 1.74e-10

Representative NCPP and USPP calculations also agree in forces and stresses as SCF convergence is tightened. Results below compare LOBPCG with Dav_Subspace at scf_thr=1e-12; force and stress entries are maximum component differences.

System Device Energy difference (eV/cell) Force difference (eV/Å) Stress difference (kbar)
Si₂ (NC) CPU 1.137e-13 8.480e-08 5.122e-05
Si₂ (NC) GPU 9.948e-13 8.480e-08 5.122e-05
NaCl (USPP) CPU 1.819e-12 1.109e-05 3.119e-03
NaCl (USPP) GPU 5.111e-10 1.114e-05 1.108e-02

These systems derive from tests/01_PW.

Performance

On this suite, LOBPCG is competitive with Dav_Subspace on CPU and substantially faster overall than CG and DAV. Dav_Subspace remains faster overall on GPU.

CPU: 28 MPI ranks with diag_subspace=2 (ScaLAPACK as inner eigensolver); GPU: one rank on one RTX 3090. LOBPCG uses hard locking, zero extra bands. CPU plots cover ten systems; GPU plots cover the nine systems where 216Si runs out of device memory.

CPU

Wall time:
cpu_ds2_total

Relative:
cpu_ds2_relative

GPU

The GPU version has not been optimized so far. Dav_Subspace remains the best choice on GPU.
gpu_total

Optional locking and extra bands

Soft locking accelerates some calculations, for example GaAs and BaTiO₃ as demonstrated below. A small buffer (extra bands, which usually takes fewer steps to converge but is more expensive per iteration) also reduced GPU time for 27Fe in the tested run. Benefits depend on the system and device; hard locking with zero extra bands remains the default and optimum configurations for most systems.

locking_buffer_effects

Memory

LOBPCG reduces memory use relative to Dav_Subspace on larger systems, making it useful when memory capacity matters. Examples use sampled peak CPU host memory (PSS, diag_subspace=2) and GPU device memory.

System Memory Saved (GiB) Reduction
216Si CPU host 7.92 17.4%
LiNMC CPU host 4.45 19.5%
27Fe CPU host 1.22 9.4%
LiNMC GPU device 1.15 12.1%

Verification and integration

  • Added unit tests for standard/generalized problems, locking and extra bands, CPU/GPU execution, MPI, and INPUT options.
  • Added the scf_lobpcg and scf_lobpcg_sca integration cases for complete SCF calculations with LAPACK and ScaLAPACK.
  • Representative SCF validation confirms NC, USPP, and spinor/SOC support on CPU and GPU, alongside the full benchmark suite of various systems.

Implement a generalized LOBPCG iteration for CPU and GPU with hard/soft
locking, optional extra bands, and reusable Rayleigh-Ritz workspaces.
Integrate the solver into HSolverPW and INPUT metadata, with focused unit
tests and CPU LAPACK/ScaLAPACK SCF cases.
@Cstandardlib
Cstandardlib marked this pull request as ready for review November 21, 2026 19:27
@Cstandardlib Cstandardlib added GPU & DCU & HPC GPU and DCU and HPC related any issues Features Needed The features are indeed needed, and developers should have sophisticated knowledge Diago Issues related to diagonalizaiton methods labels Nov 21, 2026

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review found issues in iteration-budget handling and ELPA edge cases. No builds, numerical/MPI/GPU runs or CI diagnostics were performed; the reported validation remains author-reported.

Non-blocking: Please add the tested commit, build options, exact test/SCF commands and stable case/input identifiers behind the reported results, distinguishing CPU, CUDA and ROCm coverage. Relevant logs would make the accuracy, performance and memory claims reproducible.

Comment thread source/source_hsolver/hsolver_pw.cpp Resolved
Comment thread source/source_hsolver/hsolver_pw.cpp Resolved
Comment thread source/source_hsolver/diago_lobpcg.cpp Resolved
Comment thread source/source_hsolver/rr_solver.cpp Resolved
Comment thread source/source_hsolver/diago_lobpcg.cpp Resolved
Use a small float buffer and remove near-full-space and large dense stress cases. Check convergence, reference eigenvalues, orthogonality and padding without imposing a separate residual stopping criterion.
Pass pw_diag_nmax through the common PW setup for all calculation modes
and solvers. Reject nonpositive budgets in INPUT validation.

Continue electronic SCF after LOBPCG budget exhaustion in relax,
cell-relax and MD, while retaining an error for unconverged NSCF solves.

Cover budget propagation across methods/modes, caller exit behavior,
and positive INPUT values with focused functional tests.

Validation: MODULE_HSOLVER_pw and MODULE_IO_lobpcg_input pass
(10 unit tests). Staged governance check has no errors.

Documentation: the INPUT domain metadata supplies the generated positive
value constraint; no separate generated documentation needs editing.

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incremental static review of 42c203bd..14e8c86f found two gaps in numerical test coverage. No runtime checks were performed.

Reject lobpcg with diag_subspace=1 during INPUT validation and report
supported values 0/2 to direct callers. Remove the LOBPCG ELPA adapter
and its test dependencies; defer ELPA support to a follow-up PR.

Keep public ELPA and Davidson implementations unchanged. Update INPUT
metadata, CLI help and generated documentation descriptions to state
the supported combinations, including CUDA cuSOLVER for option 0.

Validation: 8 CPU and 4 GPU CTest targets pass. CLI --check-input accepts
LOBPCG 0/2, rejects LOBPCG 1, and accepts Dav_Subspace 0/1/2. Generated
input-main and parameter help were checked. Governance has no errors.

Documentation is generated from the updated Input_Item metadata;
generated YAML and Markdown are not committed.
Use a small diagonal generalized problem with distinct eigenvalues to
check returned vector columns against their analytic eigenvectors up to
phase. Exercise the iterative path in float and double with hard and soft
locking, and include the check in the GPU MPI2 test selection.

Validation: four CPU/GPU single-process and MPI2 CTest targets pass.
Production code and existing accuracy assertions are unchanged. No user
documentation update is needed for this test-only change.
Set pw_diag_nmax=30 explicitly for MemSaver, Wannier90, and HSE NSCF
fixtures to preserve the effective budget used to generate their references.
NSCF now honors INPUT budgets, so leaving these fixtures unspecified would
select 50 and change CG restart and near-degenerate eigenvector paths.

Keep the production parameter fix, reference outputs, and comparison
thresholds unchanged.

Validation: existing Autotest harness passes all three cases on CPU
(4 MPI ranks) and MemSaver/Wannier90 on GPU (1 MPI rank), with one OpenMP
thread. Staged governance and diff whitespace checks pass.

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Source review at ee40021f confirms the revised budget handling and removal of the LOBPCG ELPA path address the corresponding earlier findings. The new diagonal fixture provides an independent eigenpair oracle. One new non-blocking test-fixture suggestion is inline, with source-status updates in the original threads. This was static review only; no builds or tests were run, and reported runtime/CI results remain unverified.

SOURCES test_diago_lobpcg_float.cpp ${LOBPCG_TEST_SOURCES}
../../source_basis/module_pw/test/test_tool.cpp)
set_tests_properties(MODULE_HSOLVER_lobpcg MODULE_HSOLVER_lobpcg_float PROPERTIES
TIMEOUT 180 ENVIRONMENT "OMP_NUM_THREADS=1;OPENBLAS_NUM_THREADS=1;MKL_NUM_THREADS=1")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this line may be too complicated, try to simplify the cmake

Resolve ESolver call-site conflicts with upstream formatting while
preserving the LOBPCG extra-band and locking constructor arguments.
Keep all other upstream changes and the existing LOBPCG implementation.

Validation: CPU/LibXC and CUDA PW executables build. Existing Autotest
checks pass for six CPU cases (LOBPCG 0/2, three NSCF regressions and
SDFT) and three GPU cases (LOBPCG, MemSaver and Wannier90).
Diff whitespace and conflict checks pass; staged governance has no errors.
No INPUT or user-facing behavior changes are introduced by the resolution.
Conjugate stored Hermitian matrix rows when accumulating distributed
output columns so the test operator applies H rather than H transpose.
Add a two-dimensional complex matrix-vector check with exact analytic
outputs, independent of the iterative eigensolver convergence criteria.

The new check fails with the previous operator. After the fix, both
MODULE_HSOLVER_lobpcg_float and its MPI2 variant pass (14 GTests total).
Existing convergence thresholds and assertions are unchanged.

No user documentation update is needed for this test-only correction.
Set thread limits once for the registered LOBPCG tests, reuse the common
MPI test helper source list, and select exit-check rank counts once.
Keep backend-specific registration and properties explicit.

CTest JSON comparison confirms unchanged commands and properties for
all 24 CPU and 31 CUDA-build HSolver test registrations, including the
existing timeouts, MPI flags, filters and device skip behavior.

Validation: six CPU and four GPU LOBPCG CTest targets pass, covering
single-process, MPI2 and fatal-exit checks. Staged governance passes.
No user documentation change is needed for this test-build refactor.
Let LOBPCG tests inherit the runtime environment and run the complete
GPU MPI2 suite without a test-name allowlist. Register exit checks next
to each device target through a small local helper, and build their MPI
launcher once before running the scenarios.

Use explicit test target names, merge adjacent GPU configuration blocks,
consolidate HSolverPW test sources, and remove an unrelated IO directory
comment. Preserve existing timeout, MPI flag and GPU skip behavior.

Validation: CPU 8/8 and GPU 4/4 CTest targets passed after the registration
cleanup. After removing the GPU MPI2 filter, the full 27-test GTest suite
passed with OMP_NUM_THREADS=2 and no OpenBLAS/MKL thread override.
CTest metadata comparisons confirm the intended environment/filter changes
and preserve the other test commands and properties. Governance and diff
whitespace checks pass. No user documentation change is needed for this
test-configuration cleanup.

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The update retains the corrected float operator fixture and expands GPU MPI selection. One test-environment suggestion is inline. This was static source review only; no builds or tests were run, and reported runtime/CI results remain unverified.

"-DMPI_PREFLAGS=${MPIEXEC_PREFLAGS}" "-DMPI_POSTFLAGS=${MPIEXEC_POSTFLAGS}"
-P "${CMAKE_CURRENT_SOURCE_DIR}/test_diago_lobpcg_exit.cmake")
set_tests_properties(${exit_name} PROPERTIES PROCESSORS ${ranks} TIMEOUT 240
SKIP_REGULAR_EXPRESSION "SKIP: no usable GPU for LOBPCG exit checks")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: Please preserve the shared thread limits for all LOBPCG test registrations. The deleted ENVIRONMENT block was the only setting for OMP_NUM_THREADS, OPENBLAS_NUM_THREADS and MKL_NUM_THREADS, and AddTest provides no replacement; callers with multithreaded defaults can therefore use extra CPU resources and face avoidable timeout risk. Restore the common assignment after the CPU/GPU, MPI and exit tests are registered.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Diago Issues related to diagonalizaiton methods Features Needed The features are indeed needed, and developers should have sophisticated knowledge GPU & DCU & HPC GPU and DCU and HPC related any issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants