Repository navigation
Description
Describe the bug
VariantType, which #12033 moved into arrow-schema, accepts any DataType::Struct as storage. supports_data_type only checks matches!(data_type, DataType::Struct(_)), so try_new, validate and Field::try_canonical_extension_type all accept structs with no metadata or value field.
This came up in apache/datafusion#26822 (for apache/datafusion#21144). Like the other canonical wrappers there, DFVariant should delegate storage validation to the Arrow extension type. Today a real check means building a VariantArray, which pulls in parquet-variant-compute.
To Reproduce
On main at 9c94861, arrow-schema with the canonical_extension_types feature:
use arrow_schema::extension::{ExtensionType, VariantType};
use arrow_schema::{DataType, Field};
let invalid = DataType::Struct(vec![Field::new("unrelated", DataType::Int32, false)].into());
// All succeed, although this is not a Variant storage type.
assert!(VariantType::try_new(&invalid, "").is_ok());
assert!(VariantType::validate(&invalid, "").is_ok());
let field = Field::new("v", invalid, true).with_extension_type(VariantType);
assert!(field.try_canonical_extension_type().is_ok());Expected behavior
Check the top-level storage fields with the rules VariantArray::try_new already applies (binary checks, value/typed_value presence):
metadatais present and isBinary,LargeBinary,BinaryView, or a dictionary or run-end-encoded array of one of thosevalue, if present, isBinary,LargeBinaryorBinaryView- at least one of
valueandtyped_valueis present
The Parquet reader attaches VariantType to every VARIANT column (extension.rs), so the check must not reject anything VariantArray::try_new accepts. arrow-go's NewVariantType checks at the same depth.
Out of scope: nullability, typed_value types and nested shredded object groups. Those stay with VariantArray::try_new.
Additional context
@mbrobbel raised this in the #8392 review, and @alamb agreed, pointing to #8387. #8387 closed after adding Binary/LargeBinary support, without changing this check.
Willingness to contribute
I plan to contribute a fix for this bug independently
Activity
apacheGH-11938: accept canonical Variant typed values
[Variant] Move VariantType into arrow-schema canonical extensions (ap…