You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Arrow canonical extension specification for arrow.parquet.variant allows UInt8, UInt16, and UInt32 as Arrow typed_value storage types, with the
following logical Variant mappings:
Arrow typed_value type
Variant primitive type
UInt8
Int16
UInt16
Int32
UInt32
Int64
However, parquet-variant-compute currently rejects these unsigned Arrow types
when constructing or shredding a VariantArray.
There is also an Arrow/Parquet schema boundary that needs special handling:
Parquet itself supports unsigned integer logical types, but the Parquet Variant
shredding specification does not allow unsigned typed_value columns. Therefore,
canonical unsigned Arrow storage must be lowered to the corresponding signed
Parquet shredded type when writing:
Arrow typed_value
Parquet Variant shredded type
UInt8
INT32 with INTEGER(16, true)
UInt16
plain INT32
UInt32
plain INT64
To Reproduce
use arrow_schema::DataType;use parquet_variant::Variant;use parquet_variant_compute::{shred_variant,VariantArray};let input = VariantArray::from_iter([Variant::Int16(0),Variant::Int16(i16::from(u8::MAX)),]);let shredded = shred_variant(&input,&DataType::UInt8)?;
Expected behavior
VariantArray and shred_variant should accept canonical Arrow UInt8, UInt16, and UInt32 typed_value storage.
Activity