You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
DynComparator does not inspect null masks. This is by design as it isn't provided a SortOptions specifying how to order them.
Unfortunately the values array of a dictionary is permitted to contain nulls, which causes the comparator to break.
To Reproduce
The builders will never put a null in a dictionary, but if you manually construct the DictionaryArray from ArrayData it is possible.
let values = Int32Array::from_iter([None, Some(1)]).into_data();
let keys =
Int32Array::from_iter([Some(0), None])
.into_data();
let data = keys
.into_builder()
.data_type(DataType::Dictionary(
Box::new(DataType::Int32),
Box::new(DataType::Int32),
))
.child_data(vec![values])
.build()
.unwrap();
let dictionary = DictionaryArray::<Int32Type>::from(data);
If you then construct a comparator on this dictionary it will say that the two rows are not equal, when they are technically both null. If you remove the Some(1) it will panic.
Expected behavior
This should be handled correctly
Additional context
Longer term I intend for these to be deprecated and replaced with the row format #2677, and so perhaps we just defer on fixing this.
Activity
alamb commentedon Oct 27, 2022
Maybe we could simply start by generating an error
tustvold commentedon Nov 25, 2022
This same issue also impacts the ability to add support for nested types, such as structs #3187