Describe the bug
The Arrow canonical extension specification for arrow.parquet.variant allows
UInt8, UInt16, and UInt32 as Arrow typed_value storage types, with the
following logical Variant mappings:
Arrow typed_value type |
Variant primitive type |
UInt8 |
Int16 |
UInt16 |
Int32 |
UInt32 |
Int64 |
However, parquet-variant-compute currently rejects these unsigned Arrow types
when constructing or shredding a VariantArray.
There is also an Arrow/Parquet schema boundary that needs special handling:
Parquet itself supports unsigned integer logical types, but the Parquet Variant
shredding specification does not allow unsigned typed_value columns. Therefore,
canonical unsigned Arrow storage must be lowered to the corresponding signed
Parquet shredded type when writing:
Arrow typed_value |
Parquet Variant shredded type |
UInt8 |
INT32 with INTEGER(16, true) |
UInt16 |
plain INT32 |
UInt32 |
plain INT64 |
To Reproduce
use arrow_schema::DataType;
use parquet_variant::Variant;
use parquet_variant_compute::{shred_variant, VariantArray};
let input = VariantArray::from_iter([
Variant::Int16(0),
Variant::Int16(i16::from(u8::MAX)),
]);
let shredded = shred_variant(&input, &DataType::UInt8)?;
Expected behavior
VariantArray and shred_variant should accept canonical Arrow UInt8, UInt16, and UInt32 typed_value storage.
Additional context
Arrow's documentation: https://arrow.apache.org/docs/format/CanonicalExtensions.html#primitive-type-mappings
Describe the bug
The Arrow canonical extension specification for
arrow.parquet.variantallowsUInt8,UInt16, andUInt32as Arrowtyped_valuestorage types, with thefollowing logical Variant mappings:
typed_valuetypeUInt8Int16UInt16Int32UInt32Int64However,
parquet-variant-computecurrently rejects these unsigned Arrow typeswhen constructing or shredding a
VariantArray.There is also an Arrow/Parquet schema boundary that needs special handling:
Parquet itself supports unsigned integer logical types, but the Parquet Variant
shredding specification does not allow unsigned
typed_valuecolumns. Therefore,canonical unsigned Arrow storage must be lowered to the corresponding signed
Parquet shredded type when writing:
typed_valueUInt8INT32withINTEGER(16, true)UInt16INT32UInt32INT64To Reproduce
Expected behavior
VariantArrayandshred_variantshould accept canonical ArrowUInt8,UInt16, andUInt32typed_value storage.Additional context
Arrow's documentation: https://arrow.apache.org/docs/format/CanonicalExtensions.html#primitive-type-mappings