Repository navigation
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
After the great work from @hhhizzz in #8733, we will (finally) have the ability to use a Bitmask filter representation when applying filters during Parquet decode.
At the moment, the code relies on a simple threshold strategy to pick between representations
arrow-rs/parquet/src/arrow/arrow_reader/read_plan.rs
Lines 107 to 130 in 911331a
| RowSelectionPolicy::Auto { threshold, .. } => { | |
| let selection = match self.selection.as_ref() { | |
| Some(selection) => selection, | |
| None => return RowSelectionStrategy::Selectors, | |
| }; | |
| let trimmed = selection.clone().trim(); | |
| let selectors: Vec<RowSelector> = trimmed.into(); | |
| if selectors.is_empty() { | |
| return RowSelectionStrategy::Mask; | |
| } | |
| let total_rows: usize = selectors.iter().map(|s| s.row_count).sum(); | |
| let selector_count = selectors.len(); | |
| if selector_count == 0 { | |
| return RowSelectionStrategy::Mask; | |
| } | |
| if total_rows < selector_count.saturating_mul(threshold) { | |
| RowSelectionStrategy::Mask | |
| } else { | |
| RowSelectionStrategy::Selectors | |
| } | |
| } |
However, as @hhhizzz mentions in #8733 (comment)
Yes, my charts indicate that there are many rules for setting the RowSelectionStrategy, like the column type, column count, string length, and their combinations... We can create tickets and collaborate on improving these over time.
Describe the solution you'd like
I would like better heuristics for selecting between the stratgies
Describe alternatives you've considered
@hhhizzz has some good suggestions, and the charts from #8733 (comment) offer some good ideas:
For how I get the the average length to use the mask, here's some statistic, you can checkout to (https://github.com/hhhizzz/arrow-rs/tree/rowselectionempty-charts) and run
python3 dev/row_selection_analysis.pyon your local machine, this is the results on my x86 PC:One column
int32, different distribution type:![]()
![]()
![]()
![]()
Different column type:
![]()
![]()
![]()
Different column counts:
![]()
![]()
![]()
![]()
![]()
Additional context












Activity
alamb commentedon Jun 22, 2026
Context from @adriangb in #9956 (comment):
The pattern that seems to work well is:
That last case specifically applies here:
alamb commentedon Jun 22, 2026
Context from @hhhizzz in #9956 (comment)
I think the best model is multi-level adaptivity.
DataFusion has more high-level context, such as query semantics, file / row-group statistics, projected columns across the whole scan, and cross-file predicate selectivity. That makes it a good place to decide whether a
RowFiltershould be used at all, or to override the default behavior when it has stronger information.However, the part this PR is trying to handle is lower-level. The actual shape of the final
RowSelectionis only known after the Parquet reader has decoded the predicate columns for a specific row group and evaluated theRowFilter. For example, whether the result is clustered, fragmented, mostly selected, or sparse is not available to DataFusion at logical / physical planning time without duplicating Parquet reader internals.So I see the responsibilities as:
The explicit policy APIs are important because they still allow DataFusion to override the automatic choice. But I think the default
Autobehavior belongs in arrow-rs, because the information it uses is produced inside the Parquet reader during execution.[-][Parquet] Better heuristics to pick between RowSelection and Mask filter representation[/-][+][Parquet] Adaptively to pick between RowSelection and Mask filter representation[/+]fix(parquet): support mask filtering across skipped pages (#10288)
bench(parquet): add async RowSelection policy baseline (#10135)