Skip to content

perf: vectorise the GainLoss decomposition and the CustomerDecisionHierarchy Yule's Q matrix #611

Open

Description

@murray-ds

What: two analysis classes do per-row or per-pair Python work that is slow at retail scale. Both predate the current branch.

  • GainLoss: the six-way decomposition calls process_customer_group once per customer through DataFrame.apply(axis=1). With millions of customers that is millions of Python calls. The same branch logic can be written once with np.select/np.where over the diff columns. process_customer_group is public, so keep it (or deprecate it) and add a test that the vectorised result matches it.
  • CustomerDecisionHierarchy: the Yule's Q distance matrix is built in a Python loop over every product pair, calling .toarray() on a dense customer vector each time. That is O(products²) iterations over arrays as long as the customer count, so 1,000 products × 1M customers means about 500k iterations over 1M-length arrays. Sparse M @ M.T plus the row sums give the a/b/c/d counts for all pairs in one pass.

Both go against the CLAUDE.md rule to use vectorised operations instead of loops or .apply().

Tests: for each, assert the vectorised output equals the current output on the existing fixtures plus one randomised case.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions