Skip to content

Security: MODE counts through an index, FREQUENCY bins by binary search - #3198

Merged
dylanmtaylor merged 3 commits into
mainfrom
fix/sec-mode-frequency-linear
Nov 4, 2026
Merged

dylanmtaylor merged 3 commits into
mainfrom
fix/sec-mode-frequency-linear

Conversation

@dylanmtaylor

@dylanmtaylor dylanmtaylor commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

What was wrong

  • MODE / MODE.SNGL / MODE.MULT counted values by linearly searching the list of distinct numbers seen so far, for every input. Distinct computed values made this quadratic.
  • FREQUENCY scanned the sorted bin list for every data value. Data above every bin made the work data length × bin count.

What changed

  • modes keeps a hash index from each number to its count. -0 is folded into 0, since they compare equal, and NaN is never merged. Tied modes keep first-occurrence order.
  • frequency uses partition_point over the bin order it already sorts. This finds the first bin at or above the value, which is the same bin the linear scan found, including with duplicate and unsorted bins.

Blast radius

Two statistical functions in gridcraft-functions. Results are unchanged.

Checks run

  • cargo test -p gridcraft-functions (new test frequency_and_mode_scale_with_their_inputs: 200,000 data values against 200,000 bins, duplicate/unsorted bins, tie order with -0)
  • cargo test -p gridcraft-calc
  • cargo clippy -p gridcraft-functions --all-targets -- -D warnings, cargo fmt --check

Checks not run

  • cargo xtask ci as a whole (the wasm build, assets, layers and parity steps were not run locally).
  • Engine and app test suites (only the changed crate and gridcraft-calc, which calls into it, were run).

Addresses CodeRabbit security findings (CWE-407).

Summary by CodeRabbit

  • Improvements
    • MODE and MODE.MULT process large inputs more efficiently, preserve first-occurrence order for tied values, count each NaN separately, and treat positive and negative zero as the same value.
    • FREQUENCY processes large datasets with unsorted or duplicate bins more efficiently. NaN bins are excluded from bin matching, and NaN data values are placed in the overflow slot.

MODE/MODE.SNGL/MODE.MULT searched the list of distinct numbers seen so far
for every input, and FREQUENCY scanned the sorted bins for every data value,
so both were quadratic in computed arrays. MODE now keys counts by value
(-0 folded into 0, NaN never merged) and keeps first-occurrence order;
FREQUENCY finds the first bin at or above each value with partition_point
over the order it already sorts. Results are unchanged.
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

Run configuration
Commits

Reviewing files that changed from the base of the PR and between 5555191 and 4b0ab92.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

No actionable comments were generated in the recent review.

Walkthrough

MODE counting now uses a hash-map index, and FREQUENCY uses partition_point to find bins. Tests cover bin placement, NaN handling, large inputs, and mode ordering.

Changes: Descriptive statistics

Layer / File(s) Summary
MODE counting and FREQUENCY lookup
crates/functions/src/stat/descriptive.rs, crates/functions/src/stat/tests.rs
MODE indexes non-NaN values by bit pattern, combines -0.0 and 0.0, and counts each NaN separately. FREQUENCY searches non-NaN bins with partition_point and places NaN data in the overflow slot. Tests cover unsorted and duplicate bins, 200,000-element inputs, distinct values, and tied-mode order.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: aiongg, harrison-booth

Merge Risk: Minimal · previous review 55551

The changed statistical lookups are consistent with their previous results, so no merge-blocking risk is identified.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly summarizes both main changes: indexed counting for MODE and binary search for FREQUENCY. It is specific and related to the pull request objectives.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 2 functions across 2 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches
Generate docstrings
  • Commit to this branch
  • Create a new PR
Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.
Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 19c6f70 and 5555191.

Files selected for processing: 2
  • crates/functions/src/stat/descriptive.rs
  • crates/functions/src/stat/tests.rs

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 5 remain after this review.

@coderabbitai coderabbitai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Fix comments: Fix CodeRabbit comments on this PR

@dylanmtaylor

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Nov 3, 2026

Copy link
Copy Markdown

Rate Limit Exceeded

@dylanmtaylor have exceeded the limit for the number of chat messages per hour. Please wait 25 minutes and 21 seconds before sending another message.

@dylanmtaylor

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Nov 3, 2026

Copy link
Copy Markdown

Rate Limit Exceeded

@dylanmtaylor have exceeded the limit for the number of chat messages per hour. Please wait 26 minutes and 1 seconds before sending another message.

@dylanmtaylor
dylanmtaylor merged commit 065f05d into main Nov 4, 2026
8 of 9 checks passed
@dylanmtaylor
dylanmtaylor deleted the fix/sec-mode-frequency-linear branch November 4, 2026 03:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant