Improve InListExpr types and expectations

@erratic-pattern

Is your feature request related to a problem or challenge?

While reviewing this PR from @erratic-pattern

fix: InList Dictionary filter pushdown type mismatch #20962:

I found myself confused about the intended type contract for InListExpr, specifically which combinations of needle and haystack types are expected to be valid. I would have expected InListExpr to compare values of the same logical type, and ideally for the implementation comments to make that explicit.

In particular, the current logic around dictionary arrays is hard to reason about. If InListExpr intentionally works on logically equivalent types rather than identical physical Arrow types, I think that should be documented clearly. If top-level dictionary haystacks are just an implementation detail, it may be cleaner to normalize them when building the static filter so the filter always stores the underlying value representation.

Describe the solution you'd like

At a minimum, I think it would help to:

document the valid needle / haystack type combinations
explain why dictionary needles are special-cased in contains
make the tests reflect the intended contract

Describe alternatives you've considered

I think it is potentially worth changing InListExpr:

to always flatten dictionaries (only use values in the in_list and then specially handle input DictionaryArrays
Only accept the same datatype for needle and haystack

This would make the API contract easier to reason about (always the same types) but the implementation can be optimized internally as well

Additional context

No response

take

@alamb

I've looked into this and wanted to check-in before proceeding since I may miss something @alamb.

Implementation can be like this imo as in the issue description:

fn instantiate_static_filter(
    in_array: ArrayRef,
) -> Result<Arc<dyn StaticFilter + Send + Sync>> {
    // Flatten dictionary-encoded haystacks to their value type so that
    // specialized filters (e.g. Int32StaticFilter) are used instead of
    // falling through to the generic ArrayStaticFilter.
    // Dictionary needles are still handled at evaluation time via the
    // `downcast_dictionary_array!` blocks in each `contains()` implementation.
    let in_array = match in_array.data_type() {
        DataType::Dictionary(_, value_type) => cast(&in_array, value_type.as_ref())?,
        _ => in_array,
    };

We would always flatten and thus be able to use specific filters such as Int32StaticFilter etc instead of falling to ArrayStaticFilter. Then I can add comments, relevant test cases and clarify contract etc.

Then as a next step I can introduce schema validation to try_new_from_array same as try_new such that we can check type mismatches early.

WDYT @alamb ?

Then as a next step I can introduce schema validation to try_new_from_array same as try_new such that we can check type mismatches early.

WDYT @alamb ?

This sounds about right to me

The basic idea of this ticket, in my opinion, is to:

Clearly document any requirements on the types of needle and haystack
Ensure the test and code follow those requirements

The easiest behavior to reason about I think would be if needle and haystack had to be exactly the same type, which I think is similar to most of the other operators in DataFusion

However, I am not sure what the implications of enforcing that would be

@alamb

Thanks @alamb for the reply I was able to finalize it today. Can you review it when you've time I dont think it is very urgent work but it would be great to have a look from you

feat: Improve InListExpr types, flatten dict haystacks and validate i…

Navigation Menu

Repository navigation

Improve `InListExpr` types and expectations #20969

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Improve InListExpr types and expectations #20969

Description

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Activity

buraksenn commented on Mar 17, 2026

buraksenn commented on Mar 19, 2026

alamb commented on Mar 22, 2026

buraksenn commented on Apr 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Improve `InListExpr` types and expectations #20969