Skip to content

Commit 5e83a5c

Browse files
dfa1claude
andcommitted
refactor(reader): drop RLE Materialized fallbacks; close ADR 0013
Replace indicesValidity bit-scatter loop with OffsetBoolArray — lazy slice over the decoded indices validity at the chunk offset. Replace emptyArray Materialized returns with LazyConstantXxxArray(len=0). Mark ADR 0013 Accepted; update 0.8.0 rollout section with what shipped. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent db2e955 commit 5e83a5c

3 files changed

Lines changed: 21 additions & 26 deletions

File tree

docs/adr/0013-drop-materialized-fallbacks.md

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# ADR 0013: Drop Materialized fallbacks once Lazy has shipped
22

3-
- **Status:** Proposed
3+
- **Status:** Accepted
44
- **Date:** 2026-06-16
55
- **Deciders:** project maintainer
66
- **Supersedes:**
@@ -98,10 +98,15 @@ A Materialized branch may be deleted when **all** of these hold:
9898
## Rollout
9999

100100
- **0.7.x:** policy ratified, no removals yet.
101-
- **0.8.0:** remove eager paths from encodings that satisfy all four
102-
criteria today: `ZigZag`, `FoR` (long/int variants), `ALP`,
103-
`Constant`, `DateTimeParts`, `DecimalByteParts`, `Decimal`,
104-
`VarBinView`, `Chunked` containers, `RunEnd`, `Sparse`, `RLE`, `Dict`.
101+
- **0.8.0:** eager paths removed from `ZigZag` (all PTypes + broadcast),
102+
`FoR` (all integer PTypes), `ALP` (broadcast-without-patches),
103+
`Constant` (Decimal → `LazyConstantDecimalArray`; `DecimalArray` interface
104+
introduced to replace two concrete `Array` permits entries),
105+
`RunEnd` (Bool → `LazyRunEndBoolArray`), `Sparse` (Bool → `LazySparseBoolArray`),
106+
`RLE` (validity → `OffsetBoolArray`; empty → `LazyConstantXxxArray`).
107+
`Dict` encoding-level path intentionally kept eager (layout-level path is lazy via ADR 0012).
108+
`DateTimeParts`, `DecimalByteParts`, `Decimal`, `VarBinView`, `Chunked` were already
109+
fully lazy before this sweep.
105110
- **0.9.0:** revisit Decompression encodings — if a Lazy variant lands
106111
(e.g. window-decoded `Pco`) the same criteria apply.
107112

docs/compatibility.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -122,7 +122,7 @@ decoder falls into one of three shapes:
122122
| `fastlanes.bitpacked` | Materialized | Materialized | window unpacks bits |
123123
| `fastlanes.delta` | Materialized | Materialized | cumulative sum requires sequential decode |
124124
| `fastlanes.for` | Lazy | Lazy | `LazyForXxxArray` (I8/U8/I16/U16/I32/U32/I64/U64), ADR 0010 + 0013 |
125-
| `fastlanes.rle` | Materialized | Lazy | run-locating accessor possible |
125+
| `fastlanes.rle` | Lazy | Lazy | `LazyRleXxxArray`; validity → `OffsetBoolArray`; empty → `LazyConstantXxxArray`, ADR 0013 |
126126
| `vortex.patched` | Materialized | Materialized | inner is full base + chunked patches (1024-elem blocks, lane-window-sorted); per-row access requires 2 laneOffsets reads + binary search inside the chunk window, so eager scatter wins for full scans |
127127
| `vortex.variant` | Materialized | TBD | shredded child reassembly |
128128
| `vortex.onpair` | n/a | n/a | not ported |

reader/src/main/java/io/github/dfa1/vortex/reader/decode/RleEncodingDecoder.java

Lines changed: 10 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -9,16 +9,16 @@
99
import io.github.dfa1.vortex.reader.array.Array;
1010
import io.github.dfa1.vortex.reader.array.ArraySegments;
1111
import io.github.dfa1.vortex.reader.array.BoolArray;
12+
import io.github.dfa1.vortex.reader.array.LazyConstantByteArray;
13+
import io.github.dfa1.vortex.reader.array.LazyConstantIntArray;
14+
import io.github.dfa1.vortex.reader.array.LazyConstantLongArray;
15+
import io.github.dfa1.vortex.reader.array.LazyConstantShortArray;
1216
import io.github.dfa1.vortex.reader.array.LazyRleByteArray;
1317
import io.github.dfa1.vortex.reader.array.LazyRleIntArray;
1418
import io.github.dfa1.vortex.reader.array.LazyRleLongArray;
1519
import io.github.dfa1.vortex.reader.array.LazyRleShortArray;
1620
import io.github.dfa1.vortex.reader.array.MaskedArray;
17-
import io.github.dfa1.vortex.reader.array.MaterializedBoolArray;
18-
import io.github.dfa1.vortex.reader.array.MaterializedByteArray;
19-
import io.github.dfa1.vortex.reader.array.MaterializedIntArray;
20-
import io.github.dfa1.vortex.reader.array.MaterializedLongArray;
21-
import io.github.dfa1.vortex.reader.array.MaterializedShortArray;
21+
import io.github.dfa1.vortex.reader.array.OffsetBoolArray;
2222

2323
import java.io.IOException;
2424
import java.lang.foreign.MemorySegment;
@@ -113,28 +113,18 @@ public Array decode(DecodeContext ctx) {
113113
if (indicesValidity == null) {
114114
return result;
115115
}
116-
int validityBytes = (int) ((rowCount + 7) / 8);
117-
MemorySegment validityBuf = ctx.arena().allocate(validityBytes);
118-
for (long j = 0; j < rowCount; j++) {
119-
if (indicesValidity.getBoolean(offset + j)) {
120-
int byteIdx = (int) (j >>> 3);
121-
byte current = validityBuf.get(ValueLayout.JAVA_BYTE, byteIdx);
122-
validityBuf.set(ValueLayout.JAVA_BYTE, byteIdx, (byte) ((current & 0xff) | (1 << (j & 7))));
123-
}
124-
}
125-
BoolArray outputValidity = new MaterializedBoolArray(new DType.Bool(false), rowCount, validityBuf);
116+
BoolArray outputValidity = new OffsetBoolArray(new DType.Bool(false), rowCount, indicesValidity, offset);
126117
return new MaskedArray(result, outputValidity);
127118
}
128119

129120
private static Array emptyArray(DecodeContext ctx) {
130-
MemorySegment empty = ctx.arena().allocate(0);
131121
DType dt = ctx.dtype();
132122
PType ptype = ((DType.Primitive) dt).ptype();
133123
return switch (ptype) {
134-
case I64, U64 -> new MaterializedLongArray(dt, 0L, empty);
135-
case I32, U32 -> new MaterializedIntArray(dt, 0L, empty);
136-
case I16, U16 -> new MaterializedShortArray(dt, 0L, empty);
137-
case I8, U8 -> new MaterializedByteArray(dt, 0L, empty);
124+
case I64, U64 -> new LazyConstantLongArray(dt, 0L, 0L);
125+
case I32, U32 -> new LazyConstantIntArray(dt, 0L, 0);
126+
case I16, U16 -> new LazyConstantShortArray(dt, 0L, (short) 0);
127+
case I8, U8 -> new LazyConstantByteArray(dt, 0L, (byte) 0);
138128
default -> throw new VortexException(EncodingId.FASTLANES_RLE, "unsupported ptype " + ptype);
139129
};
140130
}

0 commit comments

Comments
 (0)