Repository navigation
Conversation
Retain borrowed BLACS contexts for derived distributed matrices, reject the unused raw-descriptor view path, and route serial frozen layouts through the local linear algebra backend. Add lifetime and serial-layout coverage.
Name borrowed BLACS-context factories separately from communicator-owned factories because Intel MPI defines MPI_Comm as int. Update Matrix2d and tests to use the explicit borrowed-distribution API.
ABACUS-PR-BOT
left a comment
There was a problem hiding this comment.
The factory rename resolves the integer-communicator overload collision in source. Please address the three implementation issues below.
Static source review of 25fec99: git diff --check passed. No builds or tests were run, and CI was not verified.
Address the blocking findings: preserve serial view layout, support tall/skinny matrix storage, and remove INT_MIN negation UB. Also address non-blocking findings: clarify API contracts, use named locals in conditional call arguments, make matmul tests fail on internal errors instead of passing falsely, and document view overlap constraints.
ABACUS-PR-BOT
left a comment
There was a problem hiding this comment.
The three implementation issues from my previous review are addressed in 5db95a7. One earlier cleanup remains, noted in its original thread; the additional nonblocking comment concerns an existing gather error path.
Static source review only; the full-PR and incremental git diff --check checks passed. No builds or tests were run, and CI was not verified.
| const MPI_Comm comm) | ||
| { | ||
| const std::vector<unsigned char> local_payload = make_local_payload(matrix); | ||
| const int local_count = checked_mpi_count(local_payload.size()); |
There was a problem hiding this comment.
Non-blocking: Please agree across ranks on oversized payloads before throwing here, including the equivalent path in gather_on_root(). With enough memory for packing, a 32769 x 8192 double matrix on a 2 x 1 grid with block size 32768 exceeds INT_MAX on only one rank. That rank throws instead of joining the count-exchange collective, so the peer may remain blocked if the caller catches the exception instead of aborting the job.
kirk0830
left a comment
There was a problem hiding this comment.
This PR is made by the agentic programmer since the workload is quite high for this refactor task. I also review the PR.
Presently, I have only reviewed parts of the whole PR, comments are left inline, waiting for the agent to fix and reply.
| const std::shared_ptr<const Parallel_Blkcyclic> distrib | ||
| = matrix.distribution(); | ||
| std::vector<int> metadata; | ||
| metadata.reserve(LOCAL_METADATA_SIZE + matrix.nrloc() |
There was a problem hiding this comment.
these syntax looks misleading. Create the memory space for std::vector at first, then do the element assignment would be more straightforward and still insist the concept of avoiding memory fragments and performance issue by the push-back
There was a problem hiding this comment.
Changed this helper to allocate the exact-sized metadata vector first and fill it by index in fbaf69833. The single-allocation behavior is preserved without mixing reserve and push_back, and touched vectors with known final sizes follow the same pattern.
| } | ||
|
|
||
| template <typename T> | ||
| std::vector<unsigned char> make_local_matrix_buffer(const Matrix2d<T>& matrix) |
There was a problem hiding this comment.
we are already in the context of matrix, no need to mention the matrix explicitly again. make_local_buffer, even make_locbuf would be clear enough
There was a problem hiding this comment.
Renamed this to make_local_buffer and updated all call sites in fbaf69833. The enclosing matrix context already makes the buffer kind clear.
| { | ||
| const std::vector<int> metadata = make_local_metadata(matrix); | ||
| const std::size_t nmeta = metadata.size(); | ||
| const std::size_t nmeta_bytes = nmeta * sizeof(int); |
There was a problem hiding this comment.
Renamed nmeta_bytes to nbytes_meta in fbaf69833, so the n prefix denotes a count and the unit follows it consistently.
| const std::vector<int> metadata = make_local_metadata(matrix); | ||
| const std::size_t nmeta = metadata.size(); | ||
| const std::size_t nmeta_bytes = nmeta * sizeof(int); | ||
| const std::size_t data_elements = static_cast<std::size_t>(matrix.nrloc()) |
There was a problem hiding this comment.
the size of data, not the element. element is a too high level concept here
There was a problem hiding this comment.
Renamed data_elements to ndata and documented it as the number of stored T values, explicitly distinct from a byte count (fbaf69833).
| const std::size_t nmeta_bytes = nmeta * sizeof(int); | ||
| const std::size_t data_elements = static_cast<std::size_t>(matrix.nrloc()) | ||
| * static_cast<std::size_t>(matrix.ncloc()); | ||
| const std::size_t data_bytes = data_elements * sizeof(T); |
There was a problem hiding this comment.
then the nbyte of data from the size
There was a problem hiding this comment.
The data byte count is now nbytes_data = ndata * sizeof(T) in fbaf69833, making the unit transition explicit at the definition.
| const Matrix2d<T>& matrix, | ||
| const MPI_Comm comm) | ||
| { | ||
| const std::vector<unsigned char> local_buffer |
There was a problem hiding this comment.
Renamed the serialized local payload to locbuf, with recvbuf/rxbuf for receive storage, in fbaf69833. This keeps short local names while functions and types retain descriptive names.
| #include <memory> | ||
| #include <stdexcept> | ||
|
|
||
| namespace |
There was a problem hiding this comment.
anonymous namespace, good, this prevents the external use.
There was a problem hiding this comment.
Thanks; I kept these helpers in the anonymous namespace so they retain internal linkage and are not exposed as module APIs (fbaf69833).
| int mpi_count(const std::size_t nbytes) | ||
| { | ||
| return static_cast<int>(nbytes); | ||
| } |
There was a problem hiding this comment.
the function has its name mismatching its behavior. Only a casting from size_t to int, why it is named the mpi_count?
There was a problem hiding this comment.
Merged validation and conversion into checked_mpi_count(std::size_t), which checks the native int limit and then narrows, in 0ac210ae3. Collective and reduction code now share this helper.
| std::vector<int> make_byte_displs(const std::vector<int>& nbytes) | ||
| { | ||
| std::vector<int> displs(nbytes.size(), 0); | ||
| int offset = 0; | ||
| for (std::size_t ip = 0; ip < nbytes.size(); ++ip) | ||
| { | ||
| displs[ip] = offset; | ||
| if (nbytes[ip] > std::numeric_limits<int>::max() - offset) | ||
| { | ||
| throw std::invalid_argument( | ||
| "Matrix2d collective matrix buffer size overflow"); | ||
| } | ||
| offset += nbytes[ip]; | ||
| } | ||
| return displs; | ||
| } |
There was a problem hiding this comment.
can a scan with the std::accumulate solve the same problem? this will make the codes lighter. but, the function name is not satisfying. make_byte_displs but what actually does is a kind of accumulation. Consider a better name, such as make_byte_offsets instead of the displacement
There was a problem hiding this comment.
Renamed this to make_byte_offsets and kept the explicit scan in 0ac210ae3. std::accumulate returns only a total, while std::partial_sum would still require a widened accumulator and a checked conversion before every MPI count.
| std::vector<int> allgather_nbytes(const int nbytes_local, const MPI_Comm comm) | ||
| { | ||
| const int nproc = Parallel_Common::comm_size(comm); | ||
| const int nbytes_int = static_cast<int>(sizeof(int)); | ||
| std::vector<int> recvcounts(static_cast<std::size_t>(nproc), nbytes_int); | ||
| const std::vector<int> rdispls | ||
| = make_byte_displs(recvcounts); | ||
| std::vector<int> nbytes(static_cast<std::size_t>(nproc), 0); | ||
|
|
||
| Parallel_Common::allgatherv_bytes(&nbytes_local, | ||
| nbytes_int, | ||
| nbytes.data(), | ||
| recvcounts.data(), | ||
| rdispls.data(), | ||
| comm); | ||
| return nbytes; | ||
| } |
There was a problem hiding this comment.
very confusing: is this function really needed? don't any the parallelization mechanism have this support? tuning and moving with respect to the byte is quite dangerous and confusing
There was a problem hiding this comment.
Kept the two-phase exchange because Allgatherv requires receive counts and displacements before payload transfer. The first phase now reuses the existing Parallel::allgather_int through ParaWorld, and the payload phase uses allgatherv_bytes; no MPI routine is called directly (0ac210ae3).
kirk0830
left a comment
There was a problem hiding this comment.
I have made my second review, this is also the first thorough review on the whole PR. The naming of variables/functions/namespaces are not quite satisfying, which would be risks for future devel. I will fix them soon.
| = make_byte_displs(metacounts); | ||
| const int metadata_total_bytes | ||
| = metadispls.back() + metacounts.back(); | ||
| const int nbytes_int = static_cast<int>(sizeof(int)); |
There was a problem hiding this comment.
seems this amount is always used in this file. Considering a const int W_INT = static_cast(sizeof(int)); at the head of this cpp file, which might be better.
There was a problem hiding this comment.
Promoted the repeated width to a file-level constexpr int W_INT = sizeof(int) in both the collective and reduction implementations (fbaf69833).
|
|
||
| } // namespace | ||
|
|
||
| namespace matrix2d_impl |
There was a problem hiding this comment.
it seems there is something being named inside but used outside: usually the "impl" indicates the details of implementations that the user/caller does not need to know, using the interfaces exposed is enough, and what will be in the interface functions are those implementations that may enclused/collected in the namespace impl. The interface-implementation isolation can also be seen in the opaque pointer technique. Getting some knowledge on it may help you make more proper design on the interfaces
There was a problem hiding this comment.
Added @internal backend-header documentation and kept these implementation headers outside the public matrix2d.h/matrix2d_linalg.h entry points in 0ac210ae3. This preserves the façade boundary without introducing a PIMPL for the template matrix.
| class Matrix2dView; | ||
|
|
||
| template <typename T> | ||
| class NDArray<T, 2> final |
There was a problem hiding this comment.
add an annotation about the use of "final", which is not a frequently used identifier
There was a problem hiding this comment.
Added a Doxygen explanation in fbaf69833: this rank-two specialization is final to prevent ambiguous slicing and specialization through inheritance.
| Matrix2dView<const T> view() const; | ||
|
|
||
| private: | ||
| struct EmptyMatrixTag |
There was a problem hiding this comment.
what is this? can we have any explanation on it?
There was a problem hiding this comment.
Documented EmptyMatrixTag in fbaf69833 as the internal construction tag for the documented 0 x 0 non-root placeholder returned by gather_on_root().
| } | ||
|
|
||
| template <typename U> | ||
| void assert_compatible_layout(const NDArray<U, 2>& other) const |
There was a problem hiding this comment.
why its implementation is not placed in the cpp file?
There was a problem hiding this comment.
Moved the non-template assert_compatible_layout implementation into matrix2d.cpp in 0ac210ae3. The storage initializer remains in the class definition because the templated constructors instantiate it for all supported scalar types.
| @@ -0,0 +1,129 @@ | |||
| #ifndef ABACUS_SOURCE_SOURCE_BASE_MATRIX2D_VIEW_H | |||
There was a problem hiding this comment.
no leading documentations explaing about the design, purpose, usage of the implementation in this file, bad
There was a problem hiding this comment.
Added file-level Doxygen to matrix2d_view.h in fbaf69833, covering non-owning storage, row-major host versus column-major distributed addressing, distribution lifetime, and assert() bounds behavior.
| std::shared_ptr<const Parallel_Blkcyclic> distrib_; | ||
| }; | ||
|
|
||
| #endif |
There was a problem hiding this comment.
if this is the interface exposing to the external caller/user, make a complete documentation both inline and before any public functions
There was a problem hiding this comment.
Documented the public view constructors and accessors in fbaf69833, including ownership, layout, leading dimension, distribution lifetime, and non-throwing bounds contract.
| } | ||
|
|
||
| private: | ||
| T* data_; |
There was a problem hiding this comment.
cannot the std::weak_ptr of std::vector satisfy your need?
There was a problem hiding this comment.
I kept T* plus a shared distribution handle because views must also wrap external buffers and future subviews; weak_ptr<vector<T>> would restrict storage origins and still would not guarantee that individual elements remain alive. The lifetime contract is now explicit in the header (fbaf69833).
| } | ||
| } // namespace | ||
|
|
||
| void inv(Matrix2dView<double> matrix) |
There was a problem hiding this comment.
does this function twist the matrix inversion to the linear problem solving? I mean, converting the inversion to the AX=I equation solving? This is the twist of semantics, I think that is why the numpy, scipy and pytorch they still provide such an API. User should know to always use the inversion of matrix with much are on it own rather than delegating such a numerical programming sense by us
There was a problem hiding this comment.
I verified that this path uses P?GETRF + P?GETRI, so it retains explicit inversion rather than rewriting the API as AX = I. Added the same public @warning that prefers solve() unless the inverse itself is required (3bf1d7f7f).
| return static_cast<int>(value); | ||
| } | ||
|
|
||
| int queried_workspace_size(const std::complex<double>& value) |
There was a problem hiding this comment.
use ws to make consistent with the naming of blas/lapack
There was a problem hiding this comment.
Renamed the ScaLAPACK workspaces to ws, iws, lws, and liws in fbaf69833, matching BLAS/LAPACK terminology while keeping the descriptive query helper name.
ABACUS-PR-BOT
left a comment
There was a problem hiding this comment.
The current update needs __MPI retained on both Matrix2d test targets and wider or scaled intermediates for float norms. The additional API comment narrows the axpby overlap contract.
Static source review of fbbcadb540ee3baccbaf4f7a3a4040b6aa0e7af4, including the contributor delta, merge resolution, affected callers and build/test registration; no build, runtime test or numerical probe was run.
| AddTest( | ||
| TARGET MODULE_BASE_matrix2d | ||
| LIBS parameter base device container | ||
| SOURCES test_matrix2d.cpp |
There was a problem hiding this comment.
With ENABLE_MPI=ON, this directory disables __MPI, and neither Matrix2d target opts back in. That compiles out their distributed cases and MPI_Init, while linking base built with a different Parallel_Blkcyclic layout; the 2/4-rank CTest entries therefore do not validate the MPI implementation. Please add KEEP_FEATURE_DEFINITIONS __MPI to both AddTest calls and rerun the focused MPI tests.
| real_type<T> abs2(const T& value) | ||
| { | ||
| const real_type<T> real_value = static_cast<real_type<T>>(value); | ||
| return real_value * real_value; |
There was a problem hiding this comment.
For float and complex<float>, real_type<T> also makes the intermediate squares single precision. A 1x1 matrix containing 1e20f now overflows before sqrt, and 1e-30f underflows to zero, although both Frobenius norms are representable; the previous double intermediates handled these cases. Please keep wider intermediates or use scaled sum-of-squares, then cast the final result to real_type<T> and add large/small float regressions.
| * | ||
| * All views must have the same global shape and compatible host or distributed | ||
| * layouts. The result may overlap either input because each element is read | ||
| * before it is written. |
There was a problem hiding this comment.
Non-blocking: Please restrict the overlap guarantee to views with identical element mapping. Shifted views such as left=buffer and result=buffer+1 pass the shape checks, but an early write overwrites a later input (and the OpenMP loop can race). Document other overlap as unsupported and add an exact in-place alias test.
Summary
Add a self-contained two-dimensional matrix value for the LCAO layout
migration. This is a foundation-only PR: it does not migrate call sites and
does not modify or introduce any
Parallel_Orbitalsreference.Matrix2d<T>as the rank-twoNDArrayspecialization with ownedlocal storage and an optional frozen
Parallel_Blkcyclicdistribution.including global/local indexing, ownership queries, padded storage, and
descriptor access.
reductions, gather/scatter, LAPACK, and ScaLAPACK operations.
transpose(),dagger(), andmatmul()results remain valid after thesource object is destroyed.
parallel layouts through ScaLAPACK.
reductions, collectives, and linear algebra.
Scope
Matrix2din this PR.foundation type and its tests; public ABACUS behavior is unchanged.
AGENTS.md,CONTEXT.md, ADRs,.qoder, and repository-local migration notes.Verification
Verified with
abacus-dev/1.0, GNU 11.4.0, andOMP_NUM_THREADS=1.cmake --build build-pr08 --target abacus_basic_para -j 8 cmake --build build-pr08-serial --target abacus_basic_omp -j 8 cmake --build build-pr08-cxx11 --target base -j 8 OMP_NUM_THREADS=1 ctest --test-dir build-pr08 --output-on-failure \ -R "^MODULE_BASE_matrix2d" git diff --check upstream/develop...HEAD python3 tools/03_code_analysis/agent_governance_check.py \ --base upstream/develop --head HEAD --format text[820/820].[937/937].BUILD_TESTING=OFFbase library build used-std=gnu++11andpassed
[83/83], confirming the C++11 baseline for the new code.6/6passed, including 2-rank and 4-rank tests.abacus_basic_para --versionandabacus_basic_omp --versionboth reportABACUS version v3.11.0-beta13.git diff --check: clean.headers that own
std::shared_ptr<Parallel_Blkcyclic>or exposeMatrix2dvalues directly rather than pointers.
pre-commitwas unavailable in the supplied environment (pre-commit: command not found; Python module also absent), so that hook could not be runlocally.
Auto-merge is intentionally disabled.