Skip to content

Parquet Variant rejects canonical Arrow UInt8/UInt16/UInt32 typed values #10416

Description

@HuaHuaY

Describe the bug

The Arrow canonical extension specification for arrow.parquet.variant allows
UInt8, UInt16, and UInt32 as Arrow typed_value storage types, with the
following logical Variant mappings:

Arrow typed_value type Variant primitive type
UInt8 Int16
UInt16 Int32
UInt32 Int64

However, parquet-variant-compute currently rejects these unsigned Arrow types
when constructing or shredding a VariantArray.

There is also an Arrow/Parquet schema boundary that needs special handling:
Parquet itself supports unsigned integer logical types, but the Parquet Variant
shredding specification does not allow unsigned typed_value columns. Therefore,
canonical unsigned Arrow storage must be lowered to the corresponding signed
Parquet shredded type when writing:

Arrow typed_value Parquet Variant shredded type
UInt8 INT32 with INTEGER(16, true)
UInt16 plain INT32
UInt32 plain INT64

To Reproduce

use arrow_schema::DataType;
use parquet_variant::Variant;
use parquet_variant_compute::{shred_variant, VariantArray};

let input = VariantArray::from_iter([
    Variant::Int16(0),
    Variant::Int16(i16::from(u8::MAX)),
]);

let shredded = shred_variant(&input, &DataType::UInt8)?;

Expected behavior

VariantArray and shred_variant should accept canonical Arrow UInt8, UInt16, and UInt32 typed_value storage.

Additional context

Arrow's documentation: https://arrow.apache.org/docs/format/CanonicalExtensions.html#primitive-type-mappings

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions