Skip to content

CI: use the Prescott OpenBLAS kernel for ASE tests - #8348

Merged
mohanchen merged 4 commits into
deepmodeling:developfrom
Stardust0831:ci/ase-openblas-core-selection
Nov 22, 2026
Merged

mohanchen merged 4 commits into
deepmodeling:developfrom
Stardust0831:ci/ase-openblas-core-selection

Conversation

@Stardust0831

@Stardust0831 Stardust0831 commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

The GNU CI image's OpenBLAS 0.3.20 Cooperlake kernel returns incorrect DSYMM results, causing the Gamma-only Si calculator and restart tests to abort at SCF iteration 4 with a charge integral of -17.6846.

The affected library is Ubuntu 22.04's system package libopenblas0-openmp 0.3.20+ds-1, installed by Dockerfile.gnu. In this version's DYNAMIC_ARCH builds, Cooperlake inherits SkylakeX's 4-by-8 multiplication kernel but retains 16-by-2 packing parameters. OpenBLAS PR #3631, included in 0.3.21, restores consistent kernels and parameters. The controlled builds below confirm that this upstream fix resolves the observed failure.

What's changed?

Set OPENBLAS_CORETYPE=Prescott for the entire x86-64 ASE CI job, using a conservative kernel supported by the current image. The change is four workflow lines; existing tests, numerical assertions, and ABACUS source remain unchanged. Remove the override when a corrected library in the GNU image has been verified.

Linked Issue

Related: #7794.

Unit Tests and/or Case Tests for my changes

Fresh validation used the original failing GNU image (sha256:b195875e7f3262d26eb4e07960afc0e7a193861699cd281eb08f65562880b17d) and the independently built upstream eba09c937c73 executable on AMD Ryzen 9 9950X3D.

With OPENBLAS_CORETYPE=Prescott and one OpenMP/BLAS thread:

  • Confirmed that the library selected Prescott. The focused DSYMM regression has zero error; all 30 broader DSYMM/DGEMM probe cases passed.
  • Both original TestAbacusCalculator.test_calculator_results and test_restart passed with every original assertion.
  • Complete suites passed using the following commands inside the image:
(cd interfaces/ASE_interface/abacuslite && bash xtest.sh)
# 56 passed, 2 existing skips.
(cd interfaces/ASE_interface/tests && bash xtest.sh)
# 7 passed.

Workflow YAML parsing, staged governance, and whitespace checks passed:

python3 tools/03_code_analysis/agent_governance_check.py --staged
git diff --cached --check

The original failure was isolated by replacing only DSYMM while retaining Cooperlake for other BLAS operations. A hosted Intel run also directly recorded automatic Cooperlake selection and the incorrect DSYMM result. The hosted ASE job for the fixed Prescott revision passed: 56 inline tests, 2 existing skips, and 7 integration tests.

To verify the upstream fix, OpenBLAS 0.3.20 was also rebuilt in the same image with and without only the two commits from PR #3631, including LAPACK. Both runs retained Cooperlake and used the same ABACUS executable, two MPI ranks, genelpa, and one OpenMP/BLAS thread:

  • Unpatched: the minimal DSYMM probe has absolute error 16; both original Si tests reproduce the iteration-4 charge of -17.6846.
  • Patched: the minimal error is zero, all 30 broader DSYMM/DGEMM cases pass (maximum error 2.22e-16), and both Si tests pass every original assertion.

Governance Notes

  • Existing ASE tests cover this CI environment change; no new test framework or helper is needed.
  • INPUT/docs changes: none; no documentation update required for this CI-only setting.
  • Core module impact: none.
  • Exceptions requested: none.

@Growl1234

Growl1234 commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

That adds quite a few complexity. Would it be possible to simply choose the generic kernel for GitHub CI? Btw I'm surprised that such fault can be related to system OpenBLAS...

@Stardust0831

This comment was marked as outdated.

@Stardust0831 Stardust0831 changed the title CI: validate OpenBLAS and select a compatible kernel for ASE tests CI: use the Prescott OpenBLAS kernel for ASE tests Nov 21, 2026
@Stardust0831

Copy link
Copy Markdown
Collaborator Author

@Growl1234 It's already been simplified to 4 lines

@Growl1234

Growl1234 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Thanks, looks good! Maybe, we should consider updating the images base to Ubuntu 24.04, which gets rid of the OpenBLAS-0.3.20 problem described in PR summary... (@mohanchen)

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of a6296f6a found no actionable issues in the job-wide Prescott override; the original ASE tests and numerical assertions remain unchanged.

No builds or tests were run, and the reported runtime and CI results were not independently verified.

@mohanchen mohanchen added the Compile & CICD & Docs & Dependencies Issues related to compiling ABACUS label Nov 21, 2026

@kirk0830 kirk0830 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi,

Sounds nice. If this is true, it is necessary to inform the ABACUS developers and users of updating the openblas or use a proper release.
Do you have some minimal examples to reproduce the issue you found? That would be more convincing.

Thanks,
Yike

@Growl1234

Copy link
Copy Markdown
Collaborator

I guess, one can easily reproduce this with an Ubuntu 22.04 docker image setting OPENBLAS_CORETYPE=Cooperlake variable. I will have a try...

@Growl1234

Copy link
Copy Markdown
Collaborator

Do you have some minimal examples to reproduce the issue you found? That would be more convincing.

The reproducer of OpenBLAS issue seems to be covered by early commit 741ca21...

@Growl1234

Growl1234 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

I cannot reproduce the runtime error with OPENBLAS_CORETYPE=Cooperlake in my local Ubuntu-22.04 podman as OpenBLAS falls back to the supported SkylakeX kernel on my laptop (CPU is i9-14900HX with AVX2). You can only try to reproduce it with a supported machine type. (update: I found the fallback is an OpenBLAS bug fixed in 0.3.22...)

@Growl1234

Growl1234 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

I confirmed the OpenBLAS bug through rerunning the minimal DSYMM probe script under Intel SDE in Cooper Lake mode:

env -u OPENBLAS_CORETYPE OMP_NUM_THREADS=1 OPENBLAS_VERBOSE=2 \
  sde64 -cpx -- python3 -S ./openblas-dsymm-probe.py \
  /usr/lib/x86_64-linux-gnu/openblas-openmp/libopenblas.so.0

The results are below

Ubuntu OpenBLAS Selected kernel Max absolute error Wrong entries
22.04 0.3.20 Cooperlake 16 24
22.04 0.3.20 SkylakeX 0 0
22.04 0.3.20 Haswell 0 0
24.04 0.3.26 Cooperlake 0 0
24.04 0.3.26 SkylakeX 0 0
24.04 0.3.26 Haswell 0 0

@Growl1234

Growl1234 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

The ASE testing failure can also be reproduced locally by using SDE to emulate Cooperlake instructor:

@Growl1234

Growl1234 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

#8357 is an alternative; it updates the docker image to Ubuntu 24.04 so we can fully get rid of this OpenBLAS bug in CI...

@ZhouXY-PKU ZhouXY-PKU left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job!

@mohanchen mohanchen added Bugs Bugs that only solvable with sufficient knowledge of DFT Interfaces Interfaces with other packages labels Nov 22, 2026
@mohanchen
mohanchen merged commit 31ddd31 into deepmodeling:develop Nov 22, 2026
23 checks passed
Growl1234 added a commit to Growl1234/abacus-develop that referenced this pull request Nov 22, 2026
…ing#8348)"

This reverts commit 31ddd31.

With this Docker update the CI workaround is no longer needed in the future.
mohanchen pushed a commit to Growl1234/abacus-develop that referenced this pull request Nov 22, 2026
…ing#8348)"

This reverts commit 31ddd31.

With this Docker update the CI workaround is no longer needed in the future.
Growl1234 added a commit to Growl1234/abacus-develop that referenced this pull request Nov 22, 2026
…ing#8348)"

This reverts commit 31ddd31.

With this Docker update the CI workaround is no longer needed in the future.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bugs Bugs that only solvable with sufficient knowledge of DFT Compile & CICD & Docs & Dependencies Issues related to compiling ABACUS Interfaces Interfaces with other packages

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants