Skip to content

Add list_sum expression#8676

Open
mhk197 wants to merge 4 commits into
developfrom
mk/list-sum-expr
Open

Add list_sum expression#8676
mhk197 wants to merge 4 commits into
developfrom
mk/list-sum-expr

Conversation

@mhk197

@mhk197 mhk197 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Adds a vortex.list.sum scalar function: the sum of each row's list elements, one output row per input row — the equivalent of DuckDB's list_sum() and DataFusion's array_sum().

Note that list entries that are empty or all null need to be nullified. However, GroupedAccumulator returns 0 in these cases, so we need to either:

  1. Change the behavior of GroupedAccumulator and grouped sum (or even sum) kernels or
  2. Do a second pass of list element validity and sizes, construct a mask for the sums, and apply it.

The first option is probably best but can be applied in a followup PR.

@codspeed-hq

codspeed-hq Bot commented Jul 7, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 12.58%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
❌ 1 regressed benchmark
✅ 1614 untouched benchmarks
🆕 11 new benchmarks
⏩ 42 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation chunked_varbinview_into_canonical[(1000, 10)] 169.7 µs 206.1 µs -17.65%
Simulation bitwise_not_vortex_buffer_mut[128] 273.6 ns 215.3 ns +27.1%
Simulation chunked_varbinview_canonical_into[(1000, 10)] 190.5 µs 154.6 µs +23.17%
Simulation bitwise_not_vortex_buffer_mut[1024] 333.9 ns 275.6 ns +21.17%
Simulation bitwise_not_vortex_buffer_mut[2048] 427.8 ns 369.4 ns +15.79%
🆕 Simulation fsl_sum_large N/A 65.2 ms N/A
🆕 Simulation fsl_sum_medium N/A 728.1 µs N/A
🆕 Simulation fsl_sum_small N/A 57 µs N/A
🆕 Simulation list_sum_large N/A 135.7 ms N/A
🆕 Simulation list_sum_medium N/A 1.1 ms N/A
🆕 Simulation list_sum_nullable_elements_large N/A 496 ms N/A
🆕 Simulation list_sum_nullable_elements_medium N/A 4.9 ms N/A
🆕 Simulation list_sum_small N/A 144 µs N/A
🆕 Simulation listview_sum_large N/A 108.1 ms N/A
🆕 Simulation listview_sum_medium N/A 900.2 µs N/A
🆕 Simulation listview_sum_small N/A 74 µs N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/list-sum-expr (8aad0cc) with develop (11f15a9)

Open in CodSpeed

Footnotes

  1. 42 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@mhk197 mhk197 added the changelog/feature A new feature label Jul 8, 2026
@mhk197 mhk197 changed the title Add list_sum expression Add list_sum expression Jul 8, 2026
@mhk197 mhk197 marked this pull request as ready for review July 9, 2026 03:23
@mhk197 mhk197 requested a review from a team July 9, 2026 03:24
mhk197 added 4 commits July 8, 2026 20:24
Design for a vortex.list.sum scalar function: survey of DuckDB/DataFusion/
ClickHouse/Polars semantics, reuse of the grouped-sum machinery
(GroupedAccumulator), SQL NULL-on-empty semantics for engine pushdown
parity, NaN handling via NumericalAggregateOpts, and version-gated
DataFusion integration (array_sum absent from the pinned DF 54).

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Implements the vortex.list.sum scalar function per the design doc: sums
each list's elements through the grouped aggregate machinery
(GroupedAccumulator<Sum>) so widening, overflow, and NaN semantics match
sum() exactly, then nulls out empty and all-null lists to follow SQL SUM
semantics (matching DuckDB list_sum and DataFusion array_sum). NaN
handling is controlled by NumericalAggregateOpts, defaulting to
skip_nans like sum().

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197 mhk197 force-pushed the mk/list-sum-expr branch from c5f717f to 8aad0cc Compare July 9, 2026 03:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant