perf(compute): monomorphic fold read in the dict aggregate lane

Async-profiler put 39% of fusedFilteredAggregateDict's CPU in what
looked like a non-inlined MemorySegment VarHandle chain, but
PrintInlining showed the chain force-inlined fine — the frames were
attribution artifacts. The real finding was the fold's aggregate read:
LongAggFold.accept read through NumericColumns.widenLong, which the
match-table pool evaluation also calls, so widenLong's merged receiver
profile (pool MaterializedLongArray + aggregate LazyForLongArray)
compiled the per-match read as a bimorphic guard whose lost receiver
type degraded the inner segment reads to virtual calls — one set per
matching row.

Each fold now reads through its own private copy of the widening
logic, so its call-site profile only ever sees aggregate receivers and
the read stays monomorphic, like the sum lane's. The run-loop variants
also moved to one-loop-per-method (performance-neutral, but keeps
inline decisions independent and profiles attributable per shape).

fusedFilteredAggregateDict (100M rows): 178.3 -> 36.2 ms/op, ~27x over
the original 982.5 ms and within ~10 ms of the sum lane — the honest
per-match cost of the min/max/count fold. Lesson recorded in ADR 0013:
value-read helpers shared between a cold setup path and a hot loop
poison the receiver profile.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: dfa1

TODO.md

@@ -97,13 +97,12 @@ Per-encoding gotchas:
   Encoded-domain value specialization was measured as a no-win (decode is dispatch-bound, see
   `forLoopDictEncoded`); the real dict lever — the monomorphic code-segment scan — shipped as the
   `DictFilter` lane in both kernels (`fusedFilteredSumDict` 762 → 25.5 ms/op ≈ 30×;
-  `fusedFilteredAggregateDict` 983 → 178 ms/op ≈ 5.5×, single-leaf filters and `COUNT(*)`).
+  `fusedFilteredAggregateDict` 983 → 36 ms/op ≈ 27× after the profile-guided monomorphic fold
+  read; single-leaf filters and `COUNT(*)`).
   Next: (1) the columnar transducer façade — [ADR-0019](docs/adr/0019-columnar-transducer-facade.md)
   drafted (Proposed): declarative column-bound stages compiled to one fused pass, multi-aggregate
-  folds for the Calcite boundary tier; review, then implement. (2) Optional: specialize the dict
-  aggregate lane's no-null long fold to close the 178 → 25 ms gap if the boundary tier needs it;
-  (3) extend the dict lane to multi-leaf `AND`s (dict leaf as the driving scan, residual leaves
-  tested per match).
+  folds for the Calcite boundary tier; review, then implement. (2) Extend the dict lane to
+  multi-leaf `AND`s (dict leaf as the driving scan, residual leaves tested per match).
 
 ## Encodings
 

docs/adr/0013-compute-primitives.md

@@ -87,12 +87,19 @@ backing segment — no per-element `findChunk` binary search, no broadcast-guard
 | `fusedFilteredSumDict` after (`DictFilter` lane) | 25.5 ± 2.5 ms/op |
 | `fusedFilteredAggregateDict` before (per-leaf `RowPredicate`) | 982.5 ± 30.2 ms/op |
 | `fusedFilteredAggregateDict` after (`DictFilter` lane) | 178.3 ± 4.1 ms/op |
+| `fusedFilteredAggregateDict` after profile-guided fold fix | 36.2 ± 1.1 ms/op |
 
 The `filteredSum` lane reproduces the raw-scan ceiling (≈ 30×) with no measurable kernel overhead.
 The `filteredAggregate` lane (a single comparison leaf driving the same code scan, extended to the
-full `SUM`/`MIN`/`MAX`/count fold and `COUNT(*)`) lands at ≈ 5.5× — off the pure-scan ceiling
-because the per-match fold (sum, min, max, two counters) keeps its loop body scalar; specializing
-the common no-null long-aggregate fold is a possible follow-up if the boundary tier ever needs it.
+full `SUM`/`MIN`/`MAX`/count fold and `COUNT(*)`) first landed at ≈ 5.5×; async-profiler +
+`PrintInlining` then showed the residual was NOT the fold arithmetic but a bimorphic aggregate
+read — the fold shared `NumericColumns.widenLong` with the match-table pool evaluation, so the
+merged receiver profile compiled the per-match read as a bimorphic guard with partially
+devirtualized segment reads. Giving the fold its own private read (its profile only ever sees
+aggregate receivers) brought the lane to ≈ 27×, within ≈ 10 ms of the sum lane — the honest
+per-match cost of the min/max/count fold. Lesson recorded: shared value-read helpers between a
+cold setup path and a hot loop poison the receiver profile; hot folds read through their own call
+sites.
 
 ## Context
 

docs/adr/0019-columnar-transducer-facade.md

@@ -20,10 +20,10 @@ off twice:
 2. The dict code-scan lane (`DictFilter`) showed that the wins now come from **encoding-aware
    structural dispatch**: the kernel inspects the filter column's shape (an offset slice over a
    dict view with `u8` codes) and swaps the whole loop, not the per-row callback. Measured on 100M
-   rows: 762 → 25.5 ms/op for `filteredSum`, 983 → 178 ms/op for `filteredAggregate` (the fuller
-   per-match fold — sum, min, max, two counters — keeps the aggregate lane off the pure-scan
-   ceiling). A per-row lambda (`RowPredicate`) cannot express that — the lane needs to see the
-   *description* of the filter, not a compiled `test(i)`.
+   rows: 762 → 25.5 ms/op for `filteredSum`, 983 → 36 ms/op for `filteredAggregate` (the last 5×
+   of which required a profile-guided fix: the fold's aggregate read had to become monomorphic —
+   see ADR 0013's result table). A per-row lambda (`RowPredicate`) cannot express that — the lane
+   needs to see the *description* of the filter, not a compiled `test(i)`.
 
 Meanwhile the caller side is growing. The Calcite adapter (ADR 0018) invokes `filteredAggregate`
 from the boundary-chunk tier of the aggregate push-down; each new capability (a second aggregate

performance/src/main/java/io/github/dfa1/vortex/performance/ComputeKernelBenchmark.java

@@ -255,9 +255,11 @@ public long fusedFilteredSumDict() {
     /// the return value so JMH cannot eliminate the loop.
     ///
     /// Result (2026-07-02, 100M rows): 982.5 ± 30.2 ms/op through the per-leaf accessor path;
-    /// 178.3 ± 4.1 ms/op with the `DictFilter` lane — ≈ 5.5×. The gap to [#fusedFilteredSumDict()]'s
-    /// 25.5 ms is the fuller per-match fold (sum, min, max, two counters) whose heavier loop body
-    /// stays scalar; closing it is a possible follow-up specialization.
+    /// 178.3 ± 4.1 ms/op with the `DictFilter` lane; 36.2 ± 1.1 ms/op (≈ 27×) once async-profiler +
+    /// PrintInlining exposed the fold's aggregate read as bimorphic — `NumericColumns.widenLong`
+    /// was shared with the match-table pool reads, polluting its receiver profile — and the fold
+    /// got its own monomorphic read. The remaining gap to [#fusedFilteredSumDict()]'s 25.5 ms is
+    /// the genuine per-match fold work (min, max, two counters).
     ///
     /// @return the selected row count plus the sum of `measure` where `category == 7`, folded
     ///         across the whole dataset

reader/src/main/java/io/github/dfa1/vortex/reader/compute/DictFilter.java

@@ -10,7 +10,9 @@
 import io.github.dfa1.vortex.reader.array.DictLongArray;
 import io.github.dfa1.vortex.reader.array.DoubleArray;
 import io.github.dfa1.vortex.reader.array.FloatArray;
+import io.github.dfa1.vortex.reader.array.IntArray;
 import io.github.dfa1.vortex.reader.array.LongArray;
+import io.github.dfa1.vortex.reader.array.ShortArray;
 import io.github.dfa1.vortex.reader.array.OffsetDoubleArray;
 import io.github.dfa1.vortex.reader.array.OffsetFloatArray;
 import io.github.dfa1.vortex.reader.array.OffsetIntArray;
@@ -454,6 +456,11 @@ private static long count(List<Run> runs, CodeTable table, BoolArray fVal) {
 
     /// Folds the long-domain aggregate accumulator over the code-matching rows of every run.
     ///
+    /// Each run-loop variant lives in its own method: one loop per compilation unit keeps every
+    /// loop's inline decisions independent and its profile attributable per shape. (The measured
+    /// hot-path fix was elsewhere — the fold's monomorphic aggregate read, see
+    /// [LongAggFold#read(long)] — the split itself was performance-neutral.)
+    ///
     /// @param runs  the code runs in row order
     /// @param table the lowered match table
     /// @param fold  the accumulator receiving each matching row
@@ -463,34 +470,71 @@ private static void scanRunsLong(List<Run> runs, CodeTable table, LongAggFold fo
         byte only = (byte) table.only();
         for (Run run : runs) {
             MemorySegment seg = run.seg();
-            long base = run.childBase();
-            long rowBase = run.rowBase();
-            long len = run.len();
             if (seg != null && single) {
-                for (long j = 0; j < len; j++) {
-                    if (seg.get(ValueLayout.JAVA_BYTE, base + j) == only) {
-                        fold.accept(rowBase + j);
-                    }
-                }
+                scanRunSingleLong(seg, run.childBase(), run.rowBase(), run.len(), only, fold);
             } else if (seg != null) {
-                for (long j = 0; j < len; j++) {
-                    if (match[seg.get(ValueLayout.JAVA_BYTE, base + j) & 0xFF]) {
-                        fold.accept(rowBase + j);
-                    }
-                }
+                scanRunTableLong(seg, run.childBase(), run.rowBase(), run.len(), match, fold);
             } else {
-                ByteArray child = run.child();
-                for (long j = 0; j < len; j++) {
-                    if (match[child.getByte(base + j) & 0xFF]) {
-                        fold.accept(rowBase + j);
-                    }
-                }
+                scanRunAccessorLong(run.child(), run.childBase(), run.rowBase(), run.len(), match, fold);
+            }
+        }
+    }
+
+    /// Single-match segment scan feeding the long-domain fold.
+    ///
+    /// @param seg     the run's backing segment
+    /// @param base    the run's first index within the segment
+    /// @param rowBase the run's first row in the scanned view
+    /// @param len     the run length
+    /// @param only    the single matching code
+    /// @param fold    the accumulator receiving each matching row
+    private static void scanRunSingleLong(MemorySegment seg, long base, long rowBase, long len,
+                                          byte only, LongAggFold fold) {
+        for (long j = 0; j < len; j++) {
+            if (seg.get(ValueLayout.JAVA_BYTE, base + j) == only) {
+                fold.accept(rowBase + j);
+            }
+        }
+    }
+
+    /// Table-lookup segment scan feeding the long-domain fold.
+    ///
+    /// @param seg     the run's backing segment
+    /// @param base    the run's first index within the segment
+    /// @param rowBase the run's first row in the scanned view
+    /// @param len     the run length
+    /// @param match   the lowered per-code match table
+    /// @param fold    the accumulator receiving each matching row
+    private static void scanRunTableLong(MemorySegment seg, long base, long rowBase, long len,
+                                         boolean[] match, LongAggFold fold) {
+        for (long j = 0; j < len; j++) {
+            if (match[seg.get(ValueLayout.JAVA_BYTE, base + j) & 0xFF]) {
+                fold.accept(rowBase + j);
+            }
+        }
+    }
+
+    /// Accessor-fallback scan feeding the long-domain fold (segmentless or broadcast children).
+    ///
+    /// @param child   the codes child owning the run
+    /// @param base    the run's first index within the child
+    /// @param rowBase the run's first row in the scanned view
+    /// @param len     the run length
+    /// @param match   the lowered per-code match table
+    /// @param fold    the accumulator receiving each matching row
+    private static void scanRunAccessorLong(ByteArray child, long base, long rowBase, long len,
+                                            boolean[] match, LongAggFold fold) {
+        for (long j = 0; j < len; j++) {
+            if (match[child.getByte(base + j) & 0xFF]) {
+                fold.accept(rowBase + j);
             }
         }
     }
 
     /// Folds the double-domain aggregate accumulator over the code-matching rows of every run.
     ///
+    /// One run-loop variant per method, mirroring [#scanRunsLong(List, CodeTable, LongAggFold)].
+    ///
     /// @param runs  the code runs in row order
     /// @param table the lowered match table
     /// @param fold  the accumulator receiving each matching row
@@ -500,28 +544,63 @@ private static void scanRunsDouble(List<Run> runs, CodeTable table, DoubleAggFol
         byte only = (byte) table.only();
         for (Run run : runs) {
             MemorySegment seg = run.seg();
-            long base = run.childBase();
-            long rowBase = run.rowBase();
-            long len = run.len();
             if (seg != null && single) {
-                for (long j = 0; j < len; j++) {
-                    if (seg.get(ValueLayout.JAVA_BYTE, base + j) == only) {
-                        fold.accept(rowBase + j);
-                    }
-                }
+                scanRunSingleDouble(seg, run.childBase(), run.rowBase(), run.len(), only, fold);
             } else if (seg != null) {
-                for (long j = 0; j < len; j++) {
-                    if (match[seg.get(ValueLayout.JAVA_BYTE, base + j) & 0xFF]) {
-                        fold.accept(rowBase + j);
-                    }
-                }
+                scanRunTableDouble(seg, run.childBase(), run.rowBase(), run.len(), match, fold);
             } else {
-                ByteArray child = run.child();
-                for (long j = 0; j < len; j++) {
-                    if (match[child.getByte(base + j) & 0xFF]) {
-                        fold.accept(rowBase + j);
-                    }
-                }
+                scanRunAccessorDouble(run.child(), run.childBase(), run.rowBase(), run.len(), match, fold);
+            }
+        }
+    }
+
+    /// Single-match segment scan feeding the double-domain fold.
+    ///
+    /// @param seg     the run's backing segment
+    /// @param base    the run's first index within the segment
+    /// @param rowBase the run's first row in the scanned view
+    /// @param len     the run length
+    /// @param only    the single matching code
+    /// @param fold    the accumulator receiving each matching row
+    private static void scanRunSingleDouble(MemorySegment seg, long base, long rowBase, long len,
+                                            byte only, DoubleAggFold fold) {
+        for (long j = 0; j < len; j++) {
+            if (seg.get(ValueLayout.JAVA_BYTE, base + j) == only) {
+                fold.accept(rowBase + j);
+            }
+        }
+    }
+
+    /// Table-lookup segment scan feeding the double-domain fold.
+    ///
+    /// @param seg     the run's backing segment
+    /// @param base    the run's first index within the segment
+    /// @param rowBase the run's first row in the scanned view
+    /// @param len     the run length
+    /// @param match   the lowered per-code match table
+    /// @param fold    the accumulator receiving each matching row
+    private static void scanRunTableDouble(MemorySegment seg, long base, long rowBase, long len,
+                                           boolean[] match, DoubleAggFold fold) {
+        for (long j = 0; j < len; j++) {
+            if (match[seg.get(ValueLayout.JAVA_BYTE, base + j) & 0xFF]) {
+                fold.accept(rowBase + j);
+            }
+        }
+    }
+
+    /// Accessor-fallback scan feeding the double-domain fold (segmentless or broadcast children).
+    ///
+    /// @param child   the codes child owning the run
+    /// @param base    the run's first index within the child
+    /// @param rowBase the run's first row in the scanned view
+    /// @param len     the run length
+    /// @param match   the lowered per-code match table
+    /// @param fold    the accumulator receiving each matching row
+    private static void scanRunAccessorDouble(ByteArray child, long base, long rowBase, long len,
+                                              boolean[] match, DoubleAggFold fold) {
+        for (long j = 0; j < len; j++) {
+            if (match[child.getByte(base + j) & 0xFF]) {
+                fold.accept(rowBase + j);
             }
         }
     }
@@ -563,7 +642,7 @@ void accept(long row) {
                 return;
             }
             nonNull++;
-            long v = NumericColumns.widenLong(aggData, unsigned, row);
+            long v = read(row);
             sum += v;
             if (!found) {
                 min = v;
@@ -579,6 +658,31 @@ void accept(long row) {
             }
         }
 
+        /// Reads aggregate position `row` widened to a `long` — a deliberate private copy of
+        /// [NumericColumns#widenLong(Array, boolean, long)]. The shared helper's receiver profile
+        /// mixes every caller: [#matchTable(DictShape, Predicate)] feeds it the (materialized)
+        /// dictionary pool, so the fold's hot aggregate read compiled as a bimorphic guard whose
+        /// lost receiver type degraded the inner segment reads to virtual calls per match
+        /// (async-profiler + PrintInlining, 2026-07-02). This copy's profile only ever sees
+        /// aggregate receivers, keeping the read monomorphic like the sum lane's.
+        ///
+        /// @param row the zero-based position
+        /// @return the widened aggregate value
+        private long read(long row) {
+            Array data = aggData;
+            if (data instanceof LongArray la) {
+                return la.getLong(row);
+            }
+            if (data instanceof IntArray ia) {
+                return unsigned ? (ia.getInt(row) & 0xFFFFFFFFL) : ia.getInt(row);
+            }
+            if (data instanceof ShortArray sa) {
+                return unsigned ? (sa.getShort(row) & 0xFFFFL) : sa.getShort(row);
+            }
+            ByteArray ba = (ByteArray) data;
+            return unsigned ? (ba.getByte(row) & 0xFFL) : ba.getByte(row);
+        }
+
         /// Boxes the fold exactly as the kernel's long lane does.
         ///
         /// @return the fold: selected count, non-null count, the [Long] sum, and the [Long]
@@ -626,7 +730,7 @@ void accept(long row) {
                 return;
             }
             nonNull++;
-            double v = readDouble(aggData, aggIsFloat, row);
+            double v = read(row);
             sum += v;
             if (!found) {
                 min = v;
@@ -642,6 +746,16 @@ void accept(long row) {
             }
         }
 
+        /// Reads aggregate position `row` widened to a `double` — a private copy of the shared
+        /// helper, for the same receiver-profile isolation as the long fold's read: this call
+        /// site's profile only ever sees aggregate receivers, keeping the read monomorphic.
+        ///
+        /// @param row the zero-based position
+        /// @return the aggregate value widened to a double
+        private double read(long row) {
+            return aggIsFloat ? ((FloatArray) aggData).getFloat(row) : ((DoubleArray) aggData).getDouble(row);
+        }
+
         /// Boxes the fold exactly as the kernel's double lane does — the extremes carry the
         /// column's element box ([Float] for an `f32` aggregate).
         ///