Skip to content

DynComparator/LexicographicalComparator Incorrectly Handles Dictionary Nulls #2687

Closed

Description

@tustvold

Describe the bug

DynComparator does not inspect null masks. This is by design as it isn't provided a SortOptions specifying how to order them.

Unfortunately the values array of a dictionary is permitted to contain nulls, which causes the comparator to break.

To Reproduce

The builders will never put a null in a dictionary, but if you manually construct the DictionaryArray from ArrayData it is possible.

let values = Int32Array::from_iter([None, Some(1)]).into_data();
let keys =
    Int32Array::from_iter([Some(0), None])
        .into_data();

let data = keys
    .into_builder()
    .data_type(DataType::Dictionary(
        Box::new(DataType::Int32),
        Box::new(DataType::Int32),
    ))
    .child_data(vec![values])
    .build()
    .unwrap();
let dictionary = DictionaryArray::<Int32Type>::from(data);

If you then construct a comparator on this dictionary it will say that the two rows are not equal, when they are technically both null. If you remove the Some(1) it will panic.

Expected behavior

This should be handled correctly

Additional context

Longer term I intend for these to be deprecated and replaced with the row format #2677, and so perhaps we just defer on fixing this.

Activity

  1. alamb commented on Oct 27, 2022

    @alamb
    Contributor

    Maybe we could simply start by generating an error

  2. tustvold commented on Nov 25, 2022

    @tustvold
    ContributorAuthor

    This same issue also impacts the ability to add support for nested types, such as structs #3187

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions