Skip to content

RunEndEncodingDecoder drops nullable run-values validity — silent null loss #225

Description

@dfa1

Found by Raincloud triage round 4 (#205) on uci-online-retail, customerid (u16?). Same silent-corruption class as #210 (null rows decode as plausible values), in the lazy RunEnd path.

RunEndEncodingDecoder unwraps the values child's MaskedArray and passes only the raw data to LazyRunEnd*Array, which has no slot for per-run validity:

Array valuesArr  = ctx.decodeChild(1, ctx.dtype(), numRuns);
Array valuesData = valuesArr instanceof MaskedArray m ? m.inner() : valuesArr; // strips validity
...
case I16, U16 -> new LazyRunEndShortArray(ctx.dtype(), n, (ShortArray) valuesData, endsData, offset);

Array tree (chunk 0): vortex.runend(u16?) → values: fastlanes.for(u16?) → bitpacked(u16?) validity_child bool (1843 null runs). Ground truth vs Java:

row 622    rust=None  java=12346   (= FoR reference base)
row 499785 rust=None  java=0       (= raw underlying value)
row 0      rust=17850 java=17850   OK

135,080 wrong cells. The differing fillers (12346 vs 0) confirm it's the run-values' validity being dropped (chunk1 uses vortex.primitive(u16?) with its own validity child — same drop).

Fix direction: LazyRunEnd*Array must carry the run-values' validity and mark expanded rows null (a null run → all its rows null), mirroring the #210 approach; wrap the result in MaskedArray when the values child carried validity.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions