Skip to content

[Parquet] Adaptively to pick between RowSelection and Mask filter representation #8846

Open

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
After the great work from @hhhizzz in #8733, we will (finally) have the ability to use a Bitmask filter representation when applying filters during Parquet decode.

At the moment, the code relies on a simple threshold strategy to pick between representations

RowSelectionPolicy::Auto { threshold, .. } => {
let selection = match self.selection.as_ref() {
Some(selection) => selection,
None => return RowSelectionStrategy::Selectors,
};
let trimmed = selection.clone().trim();
let selectors: Vec<RowSelector> = trimmed.into();
if selectors.is_empty() {
return RowSelectionStrategy::Mask;
}
let total_rows: usize = selectors.iter().map(|s| s.row_count).sum();
let selector_count = selectors.len();
if selector_count == 0 {
return RowSelectionStrategy::Mask;
}
if total_rows < selector_count.saturating_mul(threshold) {
RowSelectionStrategy::Mask
} else {
RowSelectionStrategy::Selectors
}
}

However, as @hhhizzz mentions in #8733 (comment)

Yes, my charts indicate that there are many rules for setting the RowSelectionStrategy, like the column type, column count, string length, and their combinations... We can create tickets and collaborate on improving these over time.

Describe the solution you'd like
I would like better heuristics for selecting between the stratgies

Describe alternatives you've considered
@hhhizzz has some good suggestions, and the charts from #8733 (comment) offer some good ideas:

For how I get the the average length to use the mask, here's some statistic, you can checkout to (https://github.com/hhhizzz/arrow-rs/tree/rowselectionempty-charts) and run python3 dev/row_selection_analysis.py on your local machine, this is the results on my x86 PC:

One column int32, different distribution type:

scenario-dense80-dense80 scenario-sparse20-sparse20 scenario-spread50-spread50 scenario-uniform50-uniform50

Different column type:

dtype-int32-uniform50 dtype-utf8view-uniform50 dtype-float64-uniform50

Different column counts:

columns-C02-uniform50 columns-C04-uniform50 columns-C08-uniform50 columns-C16-uniform50 columns-C32-uniform50

Additional context

Activity

  1. alamb commented on Jun 22, 2026

    @alamb
    ContributorAuthor

    Context from @adriangb in #9956 (comment):

    I think we had been planning to put more of the adaptivity at a higher level (DataFusion specifically) as it has more information about things like statistics, and cross file predicate selectivity.

    The pattern that seems to work well is:

    • Seed with cheap heuristics
    • Adapt once runtime information is available
    • Have multiple levels of adaptivity

    That last case specifically applies here:

    • Datafusion can decide what even is a row filter or not
    • arrow-rs can optimize how it applies those row filters (e.g. when it pays the price of flattening / rolling over a mask, etc.) at a more fine-grained level
  2. alamb commented on Jun 22, 2026

    @alamb
    ContributorAuthor

    Context from @hhhizzz in #9956 (comment)

    I think the best model is multi-level adaptivity.

    DataFusion has more high-level context, such as query semantics, file / row-group statistics, projected columns across the whole scan, and cross-file predicate selectivity. That makes it a good place to decide whether a RowFilter should be used at all, or to override the default behavior when it has stronger information.

    However, the part this PR is trying to handle is lower-level. The actual shape of the final RowSelection is only known after the Parquet reader has decoded the predicate columns for a specific row group and evaluated the RowFilter. For example, whether the result is clustered, fragmented, mostly selected, or sparse is not available to DataFusion at logical / physical planning time without duplicating Parquet reader internals.

    So I see the responsibilities as:

    • DataFusion decides the higher-level scan / filtering strategy.
    • arrow-rs decides how to execute a row filter once it sees the actual row-group-level selection shape and loaded page ranges.

    The explicit policy APIs are important because they still allow DataFusion to override the automatic choice. But I think the default Auto behavior belongs in arrow-rs, because the information it uses is produced inside the Parquet reader during execution.

  3. changed the title [-][Parquet] Better heuristics to pick between RowSelection and Mask filter representation[/-] [+][Parquet] Adaptively to pick between RowSelection and Mask filter representation[/+] on Jun 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions