Skip to content

feat(base): add self-contained Matrix2d storage and math - #8317

Open
kirk0830 wants to merge 13 commits into
deepmodeling:developfrom
kirk0830:pr/08-matrix2d
Open

kirk0830 wants to merge 13 commits into
deepmodeling:developfrom
kirk0830:pr/08-matrix2d

Conversation

@kirk0830

Copy link
Copy Markdown
Collaborator

Summary

Add a self-contained two-dimensional matrix value for the LCAO layout
migration. This is a foundation-only PR: it does not migrate call sites and
does not modify or introduce any Parallel_Orbitals reference.

  • Introduce Matrix2d<T> as the rank-two NDArray specialization with owned
    local storage and an optional frozen Parallel_Blkcyclic distribution.
  • Support host row-major storage and distributed local column-major blocks,
    including global/local indexing, ownership queries, padded storage, and
    descriptor access.
  • Add elementwise operations, transpose/dagger, matrix multiplication,
    reductions, gather/scatter, LAPACK, and ScaLAPACK operations.
  • Make derived distributed matrices retain the BLACS context owner, so
    transpose(), dagger(), and matmul() results remain valid after the
    source object is destroyed.
  • Route serial frozen layouts through the local LAPACK/loop backend and active
    parallel layouts through ScaLAPACK.
  • Add serial, 2-rank, and 4-rank tests for storage, layout, lifetime,
    reductions, collectives, and linear algebra.

Scope

  • No INPUT, output-format, numerical-workflow, or user-facing behavior changes.
  • No production code is migrated to Matrix2d in this PR.
  • No replacement omnibus layout object is introduced.
  • Documentation sync is not required because this PR only adds an internal
    foundation type and its tests; public ABACUS behavior is unchanged.
  • The PR-scoped diff excludes AGENTS.md, CONTEXT.md, ADRs,
    .qoder, and repository-local migration notes.

Verification

Verified with abacus-dev/1.0, GNU 11.4.0, and OMP_NUM_THREADS=1.

cmake --build build-pr08 --target abacus_basic_para -j 8
cmake --build build-pr08-serial --target abacus_basic_omp -j 8
cmake --build build-pr08-cxx11 --target base -j 8

OMP_NUM_THREADS=1 ctest --test-dir build-pr08 --output-on-failure \
  -R "^MODULE_BASE_matrix2d"

git diff --check upstream/develop...HEAD
python3 tools/03_code_analysis/agent_governance_check.py \
  --base upstream/develop --head HEAD --format text
  • MPI + LCAO + LibXC build: [820/820].
  • Serial LCAO + OpenMP + LibXC build: [937/937].
  • Non-MPI, BUILD_TESTING=OFF base library build used -std=gnu++11 and
    passed [83/83], confirming the C++11 baseline for the new code.
  • Matrix2d focused suite: 6/6 passed, including 2-rank and 4-rank tests.
  • abacus_basic_para --version and abacus_basic_omp --version both report
    ABACUS version v3.11.0-beta13.
  • git diff --check: clean.
  • Governance: no blocking findings. Header include warnings are expected for
    headers that own std::shared_ptr<Parallel_Blkcyclic> or expose Matrix2d
    values directly rather than pointers.
  • pre-commit was unavailable in the supplied environment (pre-commit: command not found; Python module also absent), so that hook could not be run
    locally.

Auto-merge is intentionally disabled.

Retain borrowed BLACS contexts for derived distributed matrices, reject the unused raw-descriptor view path, and route serial frozen layouts through the local linear algebra backend. Add lifetime and serial-layout coverage.
Name borrowed BLACS-context factories separately from communicator-owned factories because Intel MPI defines MPI_Comm as int. Update Matrix2d and tests to use the explicit borrowed-distribution API.

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The factory rename resolves the integer-communicator overload collision in source. Please address the three implementation issues below.

Static source review of 25fec99: git diff --check passed. No builds or tests were run, and CI was not verified.

Comment thread source/source_base/matrix2d.cpp Resolved
Comment thread source/source_base/matrix2d.h Resolved
Comment thread source/source_base/matrix2d.h Resolved
Comment thread source/source_base/matrix2d_linalg.h Resolved
@mohanchen mohanchen added the Refactor Refactor ABACUS codes label Nov 17, 2026
Address the blocking findings: preserve serial view layout, support tall/skinny matrix storage, and remove INT_MIN negation UB. Also address non-blocking findings: clarify API contracts, use named locals in conditional call arguments, make matmul tests fail on internal errors instead of passing falsely, and document view overlap constraints.

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The three implementation issues from my previous review are addressed in 5db95a7. One earlier cleanup remains, noted in its original thread; the additional nonblocking comment concerns an existing gather error path.

Static source review only; the full-PR and incremental git diff --check checks passed. No builds or tests were run, and CI was not verified.

const MPI_Comm comm)
{
const std::vector<unsigned char> local_payload = make_local_payload(matrix);
const int local_count = checked_mpi_count(local_payload.size());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: Please agree across ranks on oversized payloads before throwing here, including the equivalent path in gather_on_root(). With enough memory for packing, a 32769 x 8192 double matrix on a 2 x 1 grid with block size 32768 exceeds INT_MAX on only one rank. That rank throws instead of joining the count-exchange collective, so the peer may remain blocked if the caller catches the exception instead of aborting the job.

@kirk0830 kirk0830 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is made by the agentic programmer since the workload is quite high for this refactor task. I also review the PR.
Presently, I have only reviewed parts of the whole PR, comments are left inline, waiting for the agent to fix and reply.

const std::shared_ptr<const Parallel_Blkcyclic> distrib
= matrix.distribution();
std::vector<int> metadata;
metadata.reserve(LOCAL_METADATA_SIZE + matrix.nrloc()

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these syntax looks misleading. Create the memory space for std::vector at first, then do the element assignment would be more straightforward and still insist the concept of avoiding memory fragments and performance issue by the push-back

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed this helper to allocate the exact-sized metadata vector first and fill it by index in fbaf69833. The single-allocation behavior is preserved without mixing reserve and push_back, and touched vectors with known final sizes follow the same pattern.

}

template <typename T>
std::vector<unsigned char> make_local_matrix_buffer(const Matrix2d<T>& matrix)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are already in the context of matrix, no need to mention the matrix explicitly again. make_local_buffer, even make_locbuf would be clear enough

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed this to make_local_buffer and updated all call sites in fbaf69833. The enclosing matrix context already makes the buffer kind clear.

{
const std::vector<int> metadata = make_local_metadata(matrix);
const std::size_t nmeta = metadata.size();
const std::size_t nmeta_bytes = nmeta * sizeof(int);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nbytes_meta?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed nmeta_bytes to nbytes_meta in fbaf69833, so the n prefix denotes a count and the unit follows it consistently.

const std::vector<int> metadata = make_local_metadata(matrix);
const std::size_t nmeta = metadata.size();
const std::size_t nmeta_bytes = nmeta * sizeof(int);
const std::size_t data_elements = static_cast<std::size_t>(matrix.nrloc())

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the size of data, not the element. element is a too high level concept here

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed data_elements to ndata and documented it as the number of stored T values, explicitly distinct from a byte count (fbaf69833).

const std::size_t nmeta_bytes = nmeta * sizeof(int);
const std::size_t data_elements = static_cast<std::size_t>(matrix.nrloc())
* static_cast<std::size_t>(matrix.ncloc());
const std::size_t data_bytes = data_elements * sizeof(T);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then the nbyte of data from the size

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The data byte count is now nbytes_data = ndata * sizeof(T) in fbaf69833, making the unit transition explicit at the definition.

const Matrix2d<T>& matrix,
const MPI_Comm comm)
{
const std::vector<unsigned char> local_buffer

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

locbuf?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed the serialized local payload to locbuf, with recvbuf/rxbuf for receive storage, in fbaf69833. This keeps short local names while functions and types retain descriptive names.

#include <memory>
#include <stdexcept>

namespace

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

anonymous namespace, good, this prevents the external use.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks; I kept these helpers in the anonymous namespace so they retain internal linkage and are not exposed as module APIs (fbaf69833).

Comment on lines +24 to +27
int mpi_count(const std::size_t nbytes)
{
return static_cast<int>(nbytes);
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the function has its name mismatching its behavior. Only a casting from size_t to int, why it is named the mpi_count?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merged validation and conversion into checked_mpi_count(std::size_t), which checks the native int limit and then narrows, in 0ac210ae3. Collective and reduction code now share this helper.

Comment on lines +29 to +44
std::vector<int> make_byte_displs(const std::vector<int>& nbytes)
{
std::vector<int> displs(nbytes.size(), 0);
int offset = 0;
for (std::size_t ip = 0; ip < nbytes.size(); ++ip)
{
displs[ip] = offset;
if (nbytes[ip] > std::numeric_limits<int>::max() - offset)
{
throw std::invalid_argument(
"Matrix2d collective matrix buffer size overflow");
}
offset += nbytes[ip];
}
return displs;
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can a scan with the std::accumulate solve the same problem? this will make the codes lighter. but, the function name is not satisfying. make_byte_displs but what actually does is a kind of accumulation. Consider a better name, such as make_byte_offsets instead of the displacement

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed this to make_byte_offsets and kept the explicit scan in 0ac210ae3. std::accumulate returns only a total, while std::partial_sum would still require a widened accumulator and a checked conversion before every MPI count.

Comment on lines +46 to +62
std::vector<int> allgather_nbytes(const int nbytes_local, const MPI_Comm comm)
{
const int nproc = Parallel_Common::comm_size(comm);
const int nbytes_int = static_cast<int>(sizeof(int));
std::vector<int> recvcounts(static_cast<std::size_t>(nproc), nbytes_int);
const std::vector<int> rdispls
= make_byte_displs(recvcounts);
std::vector<int> nbytes(static_cast<std::size_t>(nproc), 0);

Parallel_Common::allgatherv_bytes(&nbytes_local,
nbytes_int,
nbytes.data(),
recvcounts.data(),
rdispls.data(),
comm);
return nbytes;
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very confusing: is this function really needed? don't any the parallelization mechanism have this support? tuning and moving with respect to the byte is quite dangerous and confusing

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept the two-phase exchange because Allgatherv requires receive counts and displacements before payload transfer. The first phase now reuses the existing Parallel::allgather_int through ParaWorld, and the payload phase uses allgatherv_bytes; no MPI routine is called directly (0ac210ae3).

@mohanchen
mohanchen requested a review from MrLi000001 November 20, 2026 03:36

@kirk0830 kirk0830 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have made my second review, this is also the first thorough review on the whole PR. The naming of variables/functions/namespaces are not quite satisfying, which would be risks for future devel. I will fix them soon.

= make_byte_displs(metacounts);
const int metadata_total_bytes
= metadispls.back() + metacounts.back();
const int nbytes_int = static_cast<int>(sizeof(int));

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems this amount is always used in this file. Considering a const int W_INT = static_cast(sizeof(int)); at the head of this cpp file, which might be better.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Promoted the repeated width to a file-level constexpr int W_INT = sizeof(int) in both the collective and reduction implementations (fbaf69833).


} // namespace

namespace matrix2d_impl

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it seems there is something being named inside but used outside: usually the "impl" indicates the details of implementations that the user/caller does not need to know, using the interfaces exposed is enough, and what will be in the interface functions are those implementations that may enclused/collected in the namespace impl. The interface-implementation isolation can also be seen in the opaque pointer technique. Getting some knowledge on it may help you make more proper design on the interfaces

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added @internal backend-header documentation and kept these implementation headers outside the public matrix2d.h/matrix2d_linalg.h entry points in 0ac210ae3. This preserves the façade boundary without introducing a PIMPL for the template matrix.

class Matrix2dView;

template <typename T>
class NDArray<T, 2> final

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add an annotation about the use of "final", which is not a frequently used identifier

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a Doxygen explanation in fbaf69833: this rank-two specialization is final to prevent ambiguous slicing and specialization through inheritance.

Matrix2dView<const T> view() const;

private:
struct EmptyMatrixTag

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this? can we have any explanation on it?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented EmptyMatrixTag in fbaf69833 as the internal construction tag for the documented 0 x 0 non-root placeholder returned by gather_on_root().

Comment thread source/source_base/matrix2d.h Outdated
}

template <typename U>
void assert_compatible_layout(const NDArray<U, 2>& other) const

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why its implementation is not placed in the cpp file?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved the non-template assert_compatible_layout implementation into matrix2d.cpp in 0ac210ae3. The storage initializer remains in the class definition because the templated constructors instantiate it for all supported scalar types.

@@ -0,0 +1,129 @@
#ifndef ABACUS_SOURCE_SOURCE_BASE_MATRIX2D_VIEW_H

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no leading documentations explaing about the design, purpose, usage of the implementation in this file, bad

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added file-level Doxygen to matrix2d_view.h in fbaf69833, covering non-owning storage, row-major host versus column-major distributed addressing, distribution lifetime, and assert() bounds behavior.

std::shared_ptr<const Parallel_Blkcyclic> distrib_;
};

#endif

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if this is the interface exposing to the external caller/user, make a complete documentation both inline and before any public functions

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented the public view constructors and accessors in fbaf69833, including ownership, layout, leading dimension, distribution lifetime, and non-throwing bounds contract.

}

private:
T* data_;

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cannot the std::weak_ptr of std::vector satisfy your need?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kept T* plus a shared distribution handle because views must also wrap external buffers and future subviews; weak_ptr<vector<T>> would restrict storage origins and still would not guarantee that individual elements remain alive. The lifetime contract is now explicit in the header (fbaf69833).

}
} // namespace

void inv(Matrix2dView<double> matrix)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this function twist the matrix inversion to the linear problem solving? I mean, converting the inversion to the AX=I equation solving? This is the twist of semantics, I think that is why the numpy, scipy and pytorch they still provide such an API. User should know to always use the inversion of matrix with much are on it own rather than delegating such a numerical programming sense by us

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified that this path uses P?GETRF + P?GETRI, so it retains explicit inversion rather than rewriting the API as AX = I. Added the same public @warning that prefers solve() unless the inverse itself is required (3bf1d7f7f).

return static_cast<int>(value);
}

int queried_workspace_size(const std::complex<double>& value)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use ws to make consistent with the naming of blas/lapack

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed the ScaLAPACK workspaces to ws, iws, lws, and liws in fbaf69833, matching BLAS/LAPACK terminology while keeping the descriptive query helper name.

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current update needs __MPI retained on both Matrix2d test targets and wider or scaled intermediates for float norms. The additional API comment narrows the axpby overlap contract.

Static source review of fbbcadb540ee3baccbaf4f7a3a4040b6aa0e7af4, including the contributor delta, merge resolution, affected callers and build/test registration; no build, runtime test or numerical probe was run.

AddTest(
TARGET MODULE_BASE_matrix2d
LIBS parameter base device container
SOURCES test_matrix2d.cpp

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With ENABLE_MPI=ON, this directory disables __MPI, and neither Matrix2d target opts back in. That compiles out their distributed cases and MPI_Init, while linking base built with a different Parallel_Blkcyclic layout; the 2/4-rank CTest entries therefore do not validate the MPI implementation. Please add KEEP_FEATURE_DEFINITIONS __MPI to both AddTest calls and rerun the focused MPI tests.

real_type<T> abs2(const T& value)
{
const real_type<T> real_value = static_cast<real_type<T>>(value);
return real_value * real_value;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For float and complex<float>, real_type<T> also makes the intermediate squares single precision. A 1x1 matrix containing 1e20f now overflows before sqrt, and 1e-30f underflows to zero, although both Frobenius norms are representable; the previous double intermediates handled these cases. Please keep wider intermediates or use scaled sum-of-squares, then cast the final result to real_type<T> and add large/small float regressions.

*
* All views must have the same global shape and compatible host or distributed
* layouts. The result may overlap either input because each element is read
* before it is written.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: Please restrict the overlap guarantee to views with identical element mapping. Shifted views such as left=buffer and result=buffer+1 pass the shape checks, but an early write overwrites a later input (and the OpenMP loop can race). Document other overlap as unsupported and add an exact in-place alias test.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Refactor Refactor ABACUS codes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants