v0.12.1

Null validity and real-world file compatibility are the two themes of this release. The
Raincloud conformance suite — 247 public datasets written
by the Python vortex-data bindings — exposed a systematic family of silent-corruption bugs where
nullable encoding children's validity masks were dropped during decode, causing null rows to appear
as invented values (0.0, the FoR base, empty string). All five affected encodings are patched:
fastlanes.bitpacked validity-child chain, vortex.dict (eager + layout), vortex.runend,
vortex.sparse (primitive + utf8/binary), vortex.datetimeparts, and fastlanes.alprd.
Real-world files also uncovered scan failures: mixed per-column chunk grids, nested struct layout,
FSST-compressed string-dict offsets, RLE double/float pools, narrow-integer dict pools, all-null
columns, and zone-map stats from the current Rust writer format (vortex.zoned). Unsigned integer
silent-corruption in CSV export and filter predicates is also fixed. The conformance suite itself
ships as a gate-running weekly workflow plus four per-PR JNI interop fixtures.

Fixed

• Per-zone stats from current Rust writers (vortex.zoned, vortex-jni 0.76.0) decode again. The 0.76 zoned layout replaced the legacy vortex.stats bit-set metadata with an aggregate-function spec list and dropped the per-stat truncation flags; the reader now reconstructs the stats table from that spec list (min/max/sum/null_count), so columnZoneStats and aggregate push-down work against those files instead of throwing ClassCastException. (#197)
• Scans of files whose columns use different chunk grids no longer fail with mixed per-column chunking beyond 1-vs-N is not supported. The scan planner now splits at the merged boundary grid — the sorted union of every column's chunk boundaries, matching the Rust reference — and decodes each column's covering chunk once, slicing it zero-copy to each window. This handles both nested grids (Raincloud emotions-dataset-for-nlp, where label's coarse chunks nest inside text's finer grid) and disjoint grids (uci-beijing-multi-site-air-quality, where numeric and station boundaries do not nest). Aligned N-vs-N and 1-vs-N scans keep their existing slice-free fast path. (#221)
• Hardened the vortex.zoned metadata decoder against malformed input: an attacker-controlled length varint could overflow its pos + len bounds checks and crash with NegativeArraySizeException/IndexOutOfBoundsException; the checks are now overflow-safe and unparseable metadata falls back to per-chunk stats. Decimal columns — whose Rust zone table keeps a sum field this reader cannot map — now also fall back instead of decoding a misaligned table. (#197)
• CSV export renders nested struct columns as JSON object cells ({"field":value,...}, strings JSON-escaped, null fields as JSON null, nested structs recursed) instead of throwing unsupported array type: StructArray. (#217)
• Scanning a struct whose columns include a nested struct no longer fails chunk planning. A nested vortex.struct layout column (e.g. Raincloud countries-of-the-world's data column) is now treated as a single full-range chunk source and decoded through the layout registry's new struct decoder into a StructArray, matching the Rust reference (a nested struct spans its parent's row range, not an independent chunking). schema/count/inspect already worked; plain scans and select of sibling columns now work too. CSV export of the struct column itself remains unsupported (a separate rendering limitation). (#207)
• CSV export renders unsigned integer columns (U8–U64) with their unsigned values — high-half values previously printed as two's-complement negatives (uci-wine magnesium U8 132 exported as -124), silent corruption found by the Raincloud conformance suite. (#208)
• Unsigned integer columns are handled unsigned in the remaining consumers: the CLI filter predicates (magnesium >= 130 now matches its high-half U8 rows instead of silently dropping them — the worst class of the bug: wrong query results), the TUI grid and inspector views, and the Calcite adapter (U8/U16/U32 map to the next wider signed SQL type so their full range fits; U64 stays BIGINT and fails loud rather than surfacing a negative for values ≥ 2^63, both when read and when summed). (#216)
• fastlanes.rle decodes F64/F32 value pools (LazyRleDoubleArray, LazyRleFloatArray) — files from the Python bindings RLE-encode double columns with long constant runs, which previously failed to scan with unsupported ptype F64. (#209)
• Lazy dict decode now covers I8/U8/I16/U16 value columns (DictByteArray, DictShortArray) — files from the Python bindings dict-encode narrow-integer columns, which previously failed to scan with unsupported ptype for lazy dict. (#206)
• CSV export handles all-null (DType.Null) columns as empty fields instead of throwing unsupported array type: NullArray. (#211)
• String-dict columns whose values are FSST-compressed no longer fail to scan with IndexOutOfBoundsException — the dictionary value offsets are now read at their true ptype width instead of a hardcoded 8-byte stride (uci-magic-gamma-telescope's class column decompressed to 4-byte offsets). (#215)
• Null rows no longer silently decode as values: wrapper decoders now propagate the row validity that files from the Python bindings carry deep in the encoding tree. Three representations were dropped — a trailing validity child on fastlanes.bitpacked (reached through vortex.alp/vortex.zigzag/fastlanes.for, which delegate validity to their encoded child per the Rust ValidityChild contract), dict pools with invalid slots, and dict codes with their own validity — across both the eager vortex.dict decoder and the lazy dict layout path. Found on real data: penguins and kepler exported invented values (32.1, 0.0) for thousands of null cells. (#210)
• vortex.runend propagates nullable run-values' validity: a null run now nulls every row it covers instead of expanding to a filler value (uci-online-retail customerid u16? nulls previously decoded as the FoR base). Row validity is a lazy run-end bool over the same run-ends, matching the Rust ValidityVTable<RunEnd>. (#225)
• vortex.sparse propagates nullability: a fill_value: null array nulls every unpatched position (world-energy-consumption biofuel_cons_change_pct f64? previously decoded them as 0.0), and a null patch value nulls its own position. Row validity is a lazy sparse bool whose fill is fill_value.is_valid() and whose patch bits are the patch values' validity, matching the Rust ValidityVTable<Sparse>. (#226)
• vortex.sparse over utf8/binary values now carries the same row validity as the primitive path: a fill_value: null string column nulls every unpatched position instead of rendering an empty string, and a nullable patch value nulls its own position instead of surfacing its raw bytes. The Rust ValidityVTable<Sparse> is generic over the values encoding, so VarBin reuses the identical sparse-bool validity — completing the #226 fix for string/binary columns. (#232)
• vortex.datetimeparts propagates null component rows instead of throwing DateTimeParts: null cell at index: when any part (days/seconds/subseconds) decodes to a null — as a nullable vortex.runend child now does after #225 — the reassembled timestamp row is null, unblocking scans of bi-yalelanguages and bi-euro2016. (#235)
• fastlanes.alprd now propagates left_parts validity: null float rows no longer decode as 0.0. (#234)

Added

• Integration test for null-fill Sparse and null-run RunEnd round-trip, running on every gate. (#233)
• Real-world conformance suite against the Raincloud corpus: 247 public datasets whose Vortex files are written by the Python bindings, each validated value-for-value against its Parquet sibling. scripts/hydrate-raincloud-corpus.sh hydrates any subset (cache → mirror → local build), RaincloudConformanceIntegrationTest tests whatever is hydrated against the checked-in per-dataset status matrix, and a weekly workflow runs a size-capped sweep. First triage found four reader gaps on real data, including one silent-corruption bug (unsigned integers rendered as signed). (#205)

v0.12.0

Identity gets types: encoding ids, layout ids, and column names are now validated domain
primitives (EncodingId, LayoutId, ColumnName — sealed WellKnown/Custom shapes with a
total parse, strings only at the wire boundary), layout decode becomes pluggable through
LayoutDecoder/LayoutRegistry, and the reader gains compatibility with current Rust writers'
vortex.zoned zone-map id. Field names are strict on both sides of the file boundary — the
writer refuses what the reference toolchain cannot survive (a NUL-named column SIGABRTs its
Arrow FFI), the reader rejects duplicate/blank/control names loudly instead of corrupting
silently. Plus the dictionary code-scan lane lands in both fused compute kernels (~20×/~22×,
multi-leaf AND ~11×), and Chunk columns are one order-preserving typed map. Every wire-level
claim measured against the Rust (JNI) oracle; behavior divergences documented in
docs/compatibility.md.

Added

• Compute.filteredSum(filterColumn, predicate, aggColumn) fuses a filter and a sum into a single scan — a row folds into the total only when the predicate selects it (a null filter row is excluded) and the aggregate value is non-null — with no intermediate selection bitmap. It matches a hand-written fused loop and is ~1.5× faster than the two-pass filter + sum. (57d2225b)
• Compute.filteredAggregate(chunk, filter, aggColumn) fuses a whole multi-column RowFilter (an n-ary AND of column-bound predicate leaves) and folds the selected rows' SUM/MIN/MAX/non-null count over an aggregate column in a single pass — the multi-column counterpart of filteredSum, and the row-level kernel behind the Calcite boundary-chunk aggregate push-down. A null aggregate column counts selected rows only (COUNT(*)). (2ba54888)
• core.model.ColumnName — the validated column-name domain primitive: non-blank, no control characters (the policy the writer, schema builder, and file parser all enforce — ColumnName.violation is the single source of truth). Field names are now strict on BOTH sides of the file boundary: the writer refuses blank/control names (IllegalArgumentException; NUL additionally crashes the reference toolchain's Arrow FFI), and the reader rejects files carrying them or duplicate field names (VortexException) — wire-legal is a floor, not a policy. Printable names of any shape ($$$$$, interior spaces, emoji) remain legal and round-trip against the reference implementation. (c993a355, dca815b9, d3b5b251)
• core.model.LayoutId — typed layout identity with the same sealed shape as EncodingId (WellKnown constants plus Custom; layouts are runtime-pluggable in the reference implementation). The reader now recognizes vortex.zoned, the current canonical zone-map layout id in the Rust reference, alongside the legacy vortex.stats alias it keeps writing — files from current Rust writers scan and prune correctly. (7df3a0db)
• Layout decode is pluggable: LayoutDecoder + LayoutRegistry (reader.layout) mirror the encoding registry — LayoutRegistry.builder().registerDefaults().register(custom).build() passed to the new VortexReader.open(path, readRegistry, layoutRegistry) / VortexHttpReader overloads dispatches every layout decode, container children included, through the registry. Programmatic registration only (no service file); unknown layouts fail loudly. Zone-map pruning and filtered scans recognize the built-in layouts only. (fc488d04, dd196f17)

Changed

• The CLI uber-jar is now fully self-contained for vortex.zstd files: it bundles the FFM binding's native libzstd for all six platforms (osx/linux/windows × x86_64/aarch64) via zstd-platform — previously it shipped the binding's classes but relied on a system libzstd. The library modules keep zstd optional. (1983656b)

• The zstd binding stays a two-artifact opt-in (io.github.dfa1.zstd:zstd + zstd-platform, both optional; no JNI anywhere), and touching vortex.zstd without it now fails with an actionable VortexException naming the two artifacts to add, instead of a raw NoClassDefFoundError. Default write/read paths never touch the binding (measured). (ae9f95cc)

• The little-endian ValueLayout constants moved from PTypeIO.LE_* to VortexFormat.LE_* — endianness is a property of the wire format, not of ptypes, and VortexFormat is where format facts live. Six classes that carried private copies (including reversed-name duplicates like SHORT_LE) now share the single source; nothing outside VortexFormat defines a withOrder(LITTLE_ENDIAN) layout. (c060d34f)

• Chunk.columns() returns an order-preserving SequencedMap<ColumnName, Chunk.Column> — one map instead of two parallel string-keyed maps, with each column's Array and DType traveling together in the Column carrier. column(String) stays as boundary sugar (plus a column(ColumnName) overload); iteration order is the schema/projection order, now guaranteed even for 1–2 column chunks. Typed names originate in ScanIterator from the file's already-certified schema. (f8ad15d1)

• Compute.filteredSum over a dictionary-encoded filter column is ~20× faster (best runs ~30×): the predicate is resolved against the dictionary's value pool once and the raw u8 codes are scanned directly from their backing segments, instead of decoding every row through the per-element accessor — a fused SUM(measure) WHERE category = … over 100M rows drops from ~760 ms to ~38 ms. (85e251cc)

• Compute.filteredAggregate takes the same dictionary code-scan lane (COUNT(*) included) — ~22× faster on the same workload (~980 ms → ~46 ms over 100M rows), which the Calcite WHERE-filtered aggregate push-down inherits on its boundary chunks. (145791c7, 6e6d7dd0)

• A multi-column AND filter no longer forfeits the dictionary lane: the dict-encoded leaf drives the code scan and the remaining predicates are evaluated only on its matches — SUM(…) WHERE category = 7 AND price > 500 over 100M rows drops from ~2.3 s to ~200 ms (~11×). (12e13501)

• core.model.EncodingId is now a sealed interface: the spec constants live in the nested WellKnown enum (re-exported, so EncodingId.VORTEX_FOO call sites compile unchanged) and Custom wraps any other wire string, which for the first time lets third-party EncodingDecoder/EncodingEncoder implementations declare ids outside the spec set. parse is total over non-blank ids — an unknown id yields a typed Custom instead of an empty Optional. (ea88a91b)

• reader.decode.ArrayNode is a single record carrying the typed EncodingId; the KnownArrayNode/UnknownArrayNode split and the ArrayNode.of factory are gone. Decode dispatch, the allowUnknown passthrough, and error messages are unchanged. A crafted file with a blank encoding id now fails as VortexException instead of escaping as IllegalArgumentException. (21810d7e)

• Layout and ZonedStatsSchema moved to the new reader.layout package, and Layout's misnamed String encodingId component is now LayoutId layoutId. Unknown layouts still fail loudly, now with a typed id in the error. (7df3a0db, b08ace79)

• UnknownArray.encodingId is a typed EncodingId instead of a raw string — a Custom, or a WellKnown whose decoder is not registered. (7588aa31)

v0.11.0

SQL over Vortex grows a compute layer: a Calcite WHERE-filtered SUM/COUNT/MIN/MAX is now answered from the zone-map statistics — folding the chunks the predicate fully selects without decoding them and decoding only the one or two chunks its range cuts through (ADR 0013 §6 / ADR 0018 boundary tier). On a 100-chunk file a SELECT SUM(x) WHERE id BETWEEN … answers ~12× faster than a full scan when the range is wide. Plus the security hardening of the untrusted-input parse paths (ADR 0003) and a vortex.zstd binding bump.

Added

• VortexCalcite.connect(schemaName, tables) opens a Calcite JDBC connection with the VortexSchema registered in one call, folding away the DriverManager / unwrap / getRootSchema().add(...) boilerplate. It wires the Babel SQL parser, so columns whose names are reserved words (close, open, value, year, …) are queryable unquoted — select close, open from vtx.ohlc — which columnar files routinely need; only the typed-literal keywords (date, time, timestamp, interval) still require back-tick quoting. (24b64b32)
• The Calcite aggregate push-down rule now auto-registers over a bare jdbc:calcite: connection: a VortexTable translates to a VortexTableScan whose register() installs the rules when the planner first sees it, so SELECT MIN/MAX/COUNT/SUM over a VortexSchema is answered from zone-map statistics with no caller wiring (previously the rule had to be attached to the planner by hand). AVG joins them — AggregateReduceFunctionsRule reduces it to SUM/COUNT, both of which push down, so a whole-table AVG also decodes no data segment. Column projection and WHERE chunk-skip push-down are unchanged. (24b64b32)
• The Calcite aggregate push-down rule now rewrites a whole-table SUM(col) to a single-row Values computed from the zone-map table (via VortexTable.zoneSum), so SELECT SUM(col) answers metadata-only with no data segment decoded — joining the existing MIN/MAX/COUNT push-down. SUM over an all-null (or empty) column answers the SQL NULL, and the rule abandons to a normal scan when a zone carries no usable sum (no zone map, or an overflowed zone). (24b64b32)
• A Calcite WHERE-filtered SUM/COUNT/MIN/MAX is now answered from zone-map statistics when the predicate partitions the chunks cleanly — each chunk wholly matches or wholly fails it — folding only the kept chunks' stats into a scan-free result. A predicate that cuts through a chunk (the typical selective range) still falls back to the zone-map-pruned scan. (32cc4a29)
• A WHERE-filtered aggregate whose range cuts through a chunk no longer falls back to a full scan: the interior chunks still fold from the zone-map stats, and each straddling boundary chunk is decoded on its own and reduced under a row-level filter, so SELECT SUM(volume) WHERE close BETWEEN 100 AND 200 decodes only the one or two boundary chunks instead of every surviving chunk. The push-down still abandons to the (correct) scan for an unsigned or floating-point filter column, a non-numeric SUM, or a missing zone map. (f89b5b69)
• VortexReader.decodeChunk(chunkIndex, columns) and chunkCount() decode a single chunk for a chosen subset of columns in isolation, rather than streaming the whole file — the returned Chunk owns its memory and is valid until closed. (084a0133)
• ScanIterator.columnZoneStats(column) surfaces per-zone min/max/sum/null-count from a column's vortex.stats zone-map table without decoding any data segment — the read side of aggregate push-down (ADR 0013 §6). ArrayStats gains a sum component, decoded from the zone-map table (where the Rust reference stores it too), so the Calcite adapter now answers SUM/AVG metadata-only when every zone carries a sum, falling back to a streaming scan only for columns without a zone map. (05dd9204)

Changed

• Bumped io.github.dfa1.zstd (the vortex.zstd FFM bindings, pinned by the BOM) 0.3 → 0.6, which ships smaller jars (native debug symbols stripped). (677c2cf7, 6dcdbe94, fec0a0d3)
• Bumped Apache Calcite (the SQL adapter's engine) 1.40 → 1.42. (2f9f02c6)

Security

• DType-tree and array-node decoding are now depth-capped (64, matching the layout-tree guard): a crafted or self-referential FlatBuffer surfaces as a VortexException instead of a StackOverflowError — which, being an Error, previously escaped sanitization and leaked the reader's memory-mapped Arena. (93f8d5f4, 428026d3)
• The HTTP reader validates footer segmentSpecs against the file size before any Range request is built from them, matching the local-file path. (1d8ddebc)
• vortex.zstd decode bounds-checks each frame's declared uncompressed size and overflow-checks the total before allocating, and range-checks VarBin length prefixes — a crafted payload can no longer under-allocate or read out of bounds. (2df4e3a7, adc445e8)
• The HTTP reader parses the server-controlled Content-Range header and slices the tail buffer defensively, so a malformed response yields a VortexException rather than a raw NumberFormatException/IndexOutOfBoundsException. (feac99b7)

v0.10.0

A vortex.zstd overhaul: compression now runs through FFM bindings to the native libzstd, gaining framed (sliceable) payloads, nullable-column support, and shared-dictionary decode. Alongside it, zone-map pruning is fixed to compare in the column's type domain.

Added

• DType.isUnsigned() — true for the unsigned integer primitives (U8–U64), false otherwise. (#159)
• The vortex.zstd encoder now writes nullable columns (primitive and utf8/binary): null positions are stripped before compression and validity is emitted as a Bool child, matching the Rust reference layout. When vortex.zstd is the configured encoder, nullable primitive columns route to it directly instead of being wrapped in vortex.masked.
• new ZstdEncodingEncoder(valuesPerFrame) splits the payload into independently compressed frames of valuesPerFrame values each (one ZstdFrameMetadata per frame), letting a slice scan decompress only the frames overlapping its row range. The no-arg constructor still emits a single frame. (#170)

Changed

• The vortex.zstd encoding now compresses and decompresses through io.github.dfa1.zstd:zstd (FFM bindings to the native libzstd) instead of io.airlift:aircompressor-v3. Consumers of vortex.zstd declare a single dependency, io.github.dfa1.zstd:zstd-platform, which transitively brings the zstd binding plus the native libzstd for every supported platform (replacing the former per-platform zstd-native-<platform> artifacts).

Fixed

• vortex.zstd segments compressed with a shared (trained) dictionary now decode, via the native libzstd dictionary support, instead of being rejected. The upstream zstd.vortex compatibility fixture is read end-to-end and matches the Rust reference. (#104)
• Writing a nullable Utf8/Binary column no longer throws NullPointerException (or silently drops nulls): nullable string columns now carry their validity like nullable primitives and round-trip through vortex.masked. As a result they decode as MaskedArray (validity + values child) rather than a bare VarBinArray. (#168)
• CSV export now handles nullable columns (MaskedArray): null rows export as an empty field instead of failing with "unsupported array type for CSV export". (#168)
• Zone-map pruning now compares filter values in the column's type domain rather than by the boxed value's type. A predicate whose value is boxed at a different width (e.g. Integer on an I64 column) — or any value on a U64 column — previously pruned nothing and silently degraded to a full scan; it now prunes correctly (unsigned columns by unsigned order). As part of this, a filter value genuinely incomparable to its column (e.g. a String against a numeric column) now raises VortexException during the scan instead of silently disabling pruning — a behaviour change for callers that relied on the previous silent full scan. (#159)

v0.9.0

Two import-only breaking changes — the vortex-core types moved under io.github.dfa1.vortex.core.*, and the no-arg DType factories became constants. In return, Vortex now ships with no FlatBuffers or Protobuf runtime dependency: the .fbs/.proto schemas compile in-house to MemorySegment-native Java, dropping com.google.flatbuffers:flatbuffers-java — the last automatic-module dependency — so a named JPMS module-info is viable, and the generated wire classes are prefixed so they no longer collide on your classpath (ADR 0017).

Added

• Canonical non-nullable DType constants: DType.I8 … I64, U8 … U64, F16/F32/F64, plus BOOL, UTF8, BINARY, NULL, VARIANT; build a nullable column with DType.I64.asNullable(). (f4b22e42)

Changed

• Breaking (imports): every vortex-core type moved under io.github.dfa1.vortex.core.* — core.model (DType, PType, TimeUnit, EncodingId, ExtensionId, Time*Dtype), core.io (IoBounds, PTypeIO, VortexFormat), core.error (VortexException), core.compute (FastLanes, PrimitiveArrays), core.fbs/core.proto (wire codecs). E.g. io.github.dfa1.vortex.core.DType → io.github.dfa1.vortex.core.model.DType. (52f30c16)
• Dropped the com.google.flatbuffers:flatbuffers-java runtime dependency; the .fbs/.proto schemas compile in-house to MemorySegment-native Java, and the generated wire classes are prefixed Fbs*/Proto* so the generic names (Array, Buffer, DType, …) no longer collide on your classpath (ADR 0017). (5907302e)

Removed

• Breaking (imports): the no-arg DType factories (DType.i64(), DType.utf8(), …) — use the constants above (DType.i64() → DType.I64). DType.decimal(..)/DType.structBuilder() and the record constructors are unchanged. (f4b22e42)

v0.8.3

A Sonar-driven refactoring release: no new file-format capability, but a focused pass using SonarCloud findings to drive cleanups — dead code removed, duplication factored out, and one hot-loop micro-optimisation. Each finding was triaged (lead, not verdict) so the changes preserve behaviour and the JIT vectorisation of the hot decode loops. The interpretation framework behind this is now documented in docs/testing.md.

Performance

• FastLanes.transposeIndex / iterateIndex: replaced the per-element %// + ORDER[] indirection with permutation tables built once in a static initialiser. Faster address generation keeps more outstanding scatter misses in flight; measured 1.4×–3.4× on the transpose/undelta kernels (Apple M5, L1→DRAM working sets). The per-element decode loops stay specialised per width to preserve C2 superword vectorisation. (089b6e36, e683a634)

Removed

• Breaking (read SPI): removed EncodingDecoder.accepts(DType). It was a residual of the ADR-0001 read/write split — encode-selection semantics copied onto the decoder side, where the reader dispatches purely by EncodingId and never called it (dead since the split). EncodingEncoder.accepts is unchanged. Downstream custom EncodingDecoder implementations should delete their accepts override. (7516a544)

Changed

• Internal dedup driven by Sonar duplication findings: extracted the shared FastLanes layout + PType.bits and PrimitiveArrays.toLongs/fromLongs into core, hoisted the Materialized* array boilerplate into a shared base, factored the four BitpackedEncodingDecoder unpack loops onto one precomputed per-row schedule, added PType.isUnsigned (dropping three private copies), and deduplicated the CLI inspect plumbing and formatBytes. (ec6b9631, a74263c0, 7af0af2a, 8362a353, 87c77cc9, d8f84088, b557e573, d52e8c0c)
• Dropped dead PType switch arms in the writer's readPrimitiveElement, primitiveArrayLen, and buildTypedUniqueArray — unreachable branches flagged as uncovered. (4c6ab149, 94d2fa49, f89072a6)

Fixed

• Cleared two SonarCloud-reported bugs in the writer's SUM zone-map stat plumbing. (33798ab9)
• Suppressed java:S1172 on AbstractMaterializedArray.materialize with a reason — the arena parameter is contractual (implements Array#materialize(SegmentAllocator) for the leaf classes), not a removable unused parameter. (9b226f73)

Tests

• Filled coverage gaps surfaced by Sonar: the Materialized* materialize defaults, every SchemaCommand.formatDType arm, and the writer's global-dict cardinality fallback with U16 utf8 codes. (8741dad3, 77fad504, c2918eaa)

Docs

• docs/testing.md: new section on reading Sonar/PIT as data — the uncovered-line triage (missing-test / dead-code / defensive-by-contract), why mutation testing splits what coverage cannot, and when duplication is the deliberate price of the hot-loop rule. (8999661b)

v0.8.2

The headline is writer-side zone-map statistics: the writer now emits vortex.stats (zoned) layouts carrying per-chunk MIN/MAX, NULL_COUNT, and SUM — matching the Rust reference — so zone-map chunk pruning and aggregate push-down work on Java-written files (previously the reader could decode these stats but the writer never produced them). The release also continues the test-hardening track: the lowest-covered encoder/decoder paths are filled in, SonarCloud new-code coverage is back to 100% with the quality gate green (overall ~83%, all ratings A, zero bugs/vulnerabilities), and the build toolchain is refreshed across eight dependency bumps.

Added

• Writer: vortex.stats (zoned) layout emission, toggled by WriteOptions.enableZoneMaps. Each column is wrapped with a per-zone (one zone per chunk) statistics table; the stat set follows the Rust reference exactly. (838dba82, f2d74351)
• Writer: per-zone MIN/MAX for primitive columns including F16, extension columns (over their storage primitive), Utf8 columns (full string bounds), and dictionary-encoded columns (computed on the logical values, independent of the dict encoding). (838dba82, fb5d096a, 38ab5c51, c1198253, e51da936)
• Writer: per-zone NULL_COUNT for every column type. (135c9b37, c52d4b83, ab233b86)
• Writer: per-zone SUM for numeric primitive columns (signed → i64, unsigned → u64, float → f64; integer overflow records a null sum). Matches Rust, which sums numeric primitives and decimals but not Utf8/extension columns. (9661f554)
• Reader: RowFilter.isNull / RowFilter.isNotNull predicates with zone-map chunk pruning — IS NULL skips chunks with zero nulls, IS NOT NULL skips all-null chunks — via the per-chunk null_count. (2749b6ca)
• Reader: columnStats() aggregates null_count across a column's chunks (reported only when every chunk carries one). (cb844f23)

Changed

• Reader: the shared default HttpClient behind VortexHttpReader.open(URI, ReadRegistry) is now a package-private non-final field used purely as a unit-test seam, so the default-client overload is driven to a normal return by a mocked client instead of a live network call. Production never reassigns it. (12e46270)

Tests

• Coverage for the ten lowest-coverage encode/decode classes — ZigZagEncodingDecoder/Encoder, SequenceEncodingEncoder, VariantEncodingDecoder.dtypeFromProto (every proto→core DType arm), TimeExtensionEncoder, VarBinViewEncodingDecoder, VarBinEncodingDecoder, AlpEncodingDecoder, DateTimePartsEncodingDecoder, and DeltaEncodingDecoder — exercising guards, broadcast/constant paths, and ptype arms. (a3012d4a, c9386eda, 6c9682b8, bbb9d669, 7742ecd3)
• Writer: property-based and mutation-driven round-trips for the Delta and AlpRd encoders. (d3d245a6)
• Reader: HTTP fixtures bumped to v0.75.0 with a smoke test across all encodings; the open(URI, ReadRegistry) overload is now covered via the default-client seam. (8a1b5db2, 12e46270)
• Reader: decoder tests allocate via Arena.ofAuto() instead of the never-freed Arena.global(). (59ec2e2a)

Build

• Dependency refresh: jacoco-maven-plugin 0.8.13→0.8.15, pitest-maven 1.20.0→1.25.5, checkstyle 13.5.0→13.6.0, byte-buddy-agent 1.17.7→1.18.10, central-publishing-maven-plugin 0.10.0→0.11.0, maven-jar-plugin 3.4.1→3.5.0, maven-dependency-plugin 3.7.0→3.11.0, and actions/checkout 6→7. (dab876b7, 7b7c3580, 46659a73, 46a30be1, c6723832, 3e5fa349, c943f81b, af009116)

v0.8.1

A hardening release: no new file-format capability, but a large step up in verification rigour. Mutation testing (PIT) now guards the security-critical bounds/parse paths in core, reader, and writer at 99–100% kill rate; the build fails on any javac warning (-Xlint:all -Werror); and property-based round-trips exercise every lossless encoding plus the full cascade-selection pipeline against seeded-random inputs. The one functional addition is boxed-nullable array input on the map writeChunk path.

Added

• Writer: the map-based writeChunk path accepts boxed nullable arrays (Integer[], Long[], Double[], …) alongside primitive arrays, so columns with nulls can be written without manual validity bookkeeping. (4d18939a)

Changed

• Breaking — ExtensionEncoder.encodeAll is now abstract. The default body threw VortexException; every implementation already overrides it, so the contract now fails at compile time rather than at runtime. (2dcd69ce)
• Breaking — Estimate is now an enum { SKIP, ALWAYS_USE, COMPLETE }. The sealed interface with empty Skip/AlwaysUse records, the skip()/alwaysUse() factories, and the null "no verdict" sentinel are gone; COMPLETE is the explicit defer-to-sample-encode verdict. (c355a4bf)
• Reader cleanups: dropped a dead length < 0 blob check and a redundant offset > fileSize bounds clause, reused the shared PTypeIO little-endian layouts, and removed redundant numeric casts flagged by static analysis. (5d5fcc45, 36328285, 04cab707)

Fixed

• Writer: I8/I16 columns are excluded from the global dictionary — the reader cannot decode a narrow-int dict, so dict-encoding them produced unreadable files. (473256b1)
• Writer: WriteRegistry now iterates encoders in a deterministic order and accepts() reports honestly, fixing a non-deterministic encoder selection that broke the Windows build. (9c4ebb18)
• Reader: Pco decode now guards preDeltaN against int overflow before clamping — the subtraction is widened to long, restoring the overflow-safe path. (b7346e7c)

Build

• Zero-warning rule: -Xlint:all -Werror across all modules. The classfile lint (which only flags missing annotation class files inside third-party Arrow bytecode) is scoped off in the two Arrow-using modules only. (dab467e5, 43f6f840)
• Mutation testing (PIT): opt-in pitest profiles in core, reader, and writer, scoped to the bounds/parse classes (IoBounds, PTypeIO, WriteRegistry, ChunkImpl, …), with common config hoisted into the parent POM. (46904b24, ed8c98a1, 1200c76b, 840cc46a)
• SonarCloud: generated fbs/ and proto/ sources excluded from analysis (machine output, not hand-maintained); the deliberate per-width SIMD-loop duplication is documented in ADR 0005 rather than refactored away. Code smells dropped 857→394; coverage ~81%, all ratings A, zero bugs/vulnerabilities. (6c591293)

Tests

• Property-based lossless round-trips added for ALP (f32/f64), Delta/FoR/ZigZag/AlpRd, a bitpacked bit-width sweep, the full CascadingCompressor (every codec × cascade depth 0–3), and a Pco seeded-random distribution sweep. (dbe44aaa, a2cf3443, aede11d7, 115dd6fd, a426c1de)
• Mutation-driven test hardening lifted core/reader/writer bounds and registry classes to 99–100% kill rate. (2235499a, c9243f9a, 912fcaff)
• Integration: added Java↔Rust round-trips for vortex.patched, fastlanes.delta, and masked encodings. (13702764)
• CLI: terminal smoke tests now force class initialization so the FFM libc/kernel32 symbol resolution is actually exercised. (3f741ef7)

v0.8.0

Read and write Vortex Variant (semi-structured, JSON-shaped) columns from Java. Internally, transform encodings now decode lazily, trimming per-decode allocation. This release also hardens the reader's bounds handling on untrusted input (ADR 0003 Phase E), fixes CSV-import memory blow-ups on large files, and lifts test coverage to 80% with all Sonar ratings at A.

Added

• Writer: vortex.variant encoder. Encodes a variant column as the canonical vortex.variant container over core_storage — an all-equal column becomes a single vortex.constant, a row-varying column a vortex.chunked of per-run constants — with an optional row-aligned typed shredded child recorded in VariantMetadata.shredded_dtype. Input is VariantData(List<Scalar>) with .constant(n, v) / .shredded(...) factories. Java↔Rust (JNI) round-trip verified for constant, row-varying, and shredded columns. Scalar values only — arbitrary nested objects need vortex.parquet.variant (deferred, ADR 0014). (35da529d, e4e44980, 4566dca0)
• Reader: variant columns now decode Java-side. ConstantEncodingDecoder and ChunkedEncodingDecoder handle DType.Variant (materialising the inner-typed array); VariantEncodingDecoder wraps the result as VariantArray, exposing coreStorage() and shredded(). (76e4c741, 4566dca0)

Security

• Reader bounds hardening (ADR 0003 Phase E): untrusted offsets/lengths from file metadata now flow through a typed IoBounds helper that throws VortexException instead of a raw IndexOutOfBoundsException, and hand-rolled index guards were replaced with Objects.checkIndex. A crafted flat-segment file can no longer trip an unchecked array access during decode. (e9af80d6, 3bcd9881, a5ce8380)

Fixed

• CSV import: large files no longer OOM. The importer now streams rows in a single pass (buffering only the first chunk for schema inference) and disables the global-dictionary pass by default, which previously accumulated every distinct value in memory. (d5280ae2, 0b6784b5, 62863616)
• CLI: IoWorker.runAndAwait decremented its in-flight counter after signaling completion, so a caller reading pending() right after it returned could still see the task counted; the counter is now decremented before the await returns. The view/tui commands also close the opened VortexHandle on every error path (openOnWorker returns Optional). (95c06b1a, 27446d81)
• Reader: BoolArray.materialize masked the accumulator byte before the bit-set OR, removing a sign-promotion footgun in the packed-bitmap write. (bc8e9d4e)

Changed

• Decode shape: transform encodings now decode lazy-only. The eager Materialized*Array fallbacks were removed from vortex.zigzag (all PTypes + broadcast, cd59fefa), fastlanes.for (all integer PTypes, d7953e1f), vortex.alp (broadcast-without-patches, deab8067), vortex.constant (Decimal → LazyConstantDecimalArray, a6a9611e), vortex.runend (Bool → LazyRunEndBoolArray, 0bbcb81f), vortex.sparse (Bool → LazySparseBoolArray, db2e955b), and fastlanes.rle (validity → OffsetBoolArray, empty → LazyConstantXxxArray, 5e83a5c3). Decompression encodings (bitpacked, pco, zstd, fsst, delta, patched), the primitive base, the vortex.dict encoding-level path, and the vortex.alp patches path stay Materialized by design. See ADR 0015.
• Breaking — sealed Array permits changed. DecimalArray is now a non-sealed family interface (decimal arrays moved from implements Array to implements DecimalArray), so decimal joins the per-dtype family layer. Downstream exhaustive switch over Array must add a case DecimalArray. (a6a9611e)
• Breaking — Array API. Array.truncate(rows) renamed to Array.limited(rows) and made an abstract operation implemented by every array (composites slice their children); raw-segment access moved off the ArraySegments utility onto Array.materialize(SegmentAllocator) and Array.segmentIfPresent(). (87ab65e2, 4d9ac1f8, 332b067e, 32a35e03)
• CSV import reports progress every 10K rows instead of per-chunk. (07a056e7)

Removed

• Breaking — EmptyArray removed from the sealed Array permits. It was never emitted by the reader (empties are zero-length typed arrays in their own family) and broke the dtype→family invariant (EmptyArray(I64) was not a LongArray). Represent an empty column as a zero-length array of the appropriate family. (3a4dcdfa)

Documentation

• ADR 0016: captures vortex-arrow bridge interop options (separate module / Arrow C-Data / none); deferred until a concrete downstream need. (a6126f29)

Tests

• Test coverage raised from ~74% to 80% — the lazy/chunked/dict/run-end/sparse array families, ChunkImpl, and several decoders (DecimalEncodingDecoder, DictEncodingDecoder, ParquetImporter) reached full line + branch coverage. SonarCloud quality gate green: reliability, security, and maintainability all at A, zero bugs and vulnerabilities.

v0.7.3

Parquet ZSTD support, vortex.patched encoder, constant-encoding selection fix, Windows TUI raw-mode fix.

Added

• Parquet: ZSTD-compressed Parquet import — zstd-jni was an optional dep in hardwood and had to be declared explicitly. NYC Yellow Taxi 2024-01 (47.6 MB Parquet, 2.96 M rows × 19 cols) imports to 40.7 MB Vortex — 14% smaller than the Rust JNI reference (47 MB) thanks to the global-dict encoder catching low-cardinality F64 columns.
• Writer: vortex.patched encoder — identifies outlier values that exceed the optimal bit width, zeros them in the inner array (exposed as an open cascade child for further bitpacking), and stores their within-chunk U16 indices and raw values separately.

Fixed

• CLI: Windows TUI raw-mode — readKey now calls ReadFile directly on the kernel handle obtained via GetStdHandle instead of reading from System.in. Java's System.in goes through JVM-internal CRT wrappers that ignore SetConsoleMode, so every keypress previously required Enter before the TUI reacted.
• Writer: constant encoding skipped for single-distinct-value columns — isDictCandidate returned true for distinctCount == 1, routing all-same-value columns through the global-dict path instead of vortex.constant.

Changed

• CLI: polling loop in Terminal.readKey(Duration) extracted to KeyDecoder.nextWithTimeout(InputStream, Duration) — eliminates duplication between PosixTerminal and WindowsTerminal.

Tests

• Integration: TaxiParquetOracleVsJavaIntegrationTest — hardwood reads the taxi Parquet to a CSV (oracle); ParquetImporter → CsvExporter produces a second CSV (SUT); line-by-line diff must be zero. Proves the importer loses no data across 2.96 M rows × 19 columns.

Full changelog: https://github.com/dfa1/vortex-java/blob/main/CHANGELOG.md#0.7.3