Skip to content

Commit 3f501f4

Browse files
xudong963claude
andauthored
Skip RowFilter and page pruning for fully matched row groups (#21637)
## Which issue does this PR close? - Closes #19028. ## Rationale for this change When DataFusion evaluates a Parquet scan with filter pushdown, it uses row group statistics to determine which row groups contain matching rows. The `RowGroupAccessPlanFilter` already tracks which row groups are "fully matched" — where statistics prove that **all** rows satisfy the predicate (via `is_fully_matched`). However, this information was not propagated downstream. Even for fully matched row groups: 1. **Page index pruning** still evaluated page-level statistics (wasted work since no pages can be pruned) 2. **RowFilter evaluation** still decoded filter columns and evaluated the predicate for every row (wasted work since every row passes) This is especially costly when filter columns are expensive to decode (e.g., large strings) or when predicates are complex. Common real-world examples include time-range filters where entire row groups fall within the range, or `WHERE status != 'DELETED'` on data with no deleted rows. ## What changes are included in this PR? ### DataFusion changes (this PR) 1. **`row_group_filter.rs`**: `RowGroupAccessPlanFilter::build()` now returns `(ParquetAccessPlan, Vec<usize>)` — the access plan plus the indices of fully matched row groups. 2. **`page_filter.rs`**: `prune_plan_with_page_index()` accepts a `fully_matched_row_groups` parameter and skips page-level pruning for those row groups. 3. **`opener.rs`**: Wires fully matched row groups through the pipeline — passes them to page pruning and to the `ParquetPushDecoderBuilder` via `with_fully_matched_row_groups()`. ### Arrow-rs dependency (apache/arrow-rs#9694) The new `ArrowReaderBuilder::with_fully_matched_row_groups()` API in arrow-rs allows skipping `RowFilter` evaluation during Parquet decoding for specified row groups. This PR uses `[patch.crates-io]` pointing to the arrow-rs fork branch until that PR is merged and released. ### Benchmark Includes a criterion benchmark (`parquet_fully_matched_filter`) using `ParquetPushDecoder` directly — the same code path DataFusion's async opener uses. Dataset: 20 row groups × 50K rows, with a 1KB string payload column and predicate `x < 200` (all row groups fully matched). | Scenario | Time | vs. baseline | |----------|------|-------------| | Filter pushdown, no skip | ~43 ms | baseline | | **Filter pushdown, with skip** | **~20 ms** | **~2.2x faster** | | No pushdown at all | ~24 ms | — | ## Are these changes tested? - All 82 existing non-submodule `datafusion-datasource-parquet` tests pass (16 failures are pre-existing, caused by missing `parquet-testing` submodule) - The benchmark verifies correctness by asserting the expected row count - Clippy and fmt pass ## Are there any user-facing changes? No user-facing API changes. This is a transparent performance optimization — queries that previously worked will now be faster when row group statistics prove all rows match the predicate. ~~**Note:** This PR depends on apache/arrow-rs#9694. The `[patch.crates-io]` in `Cargo.toml` will be removed once that arrow-rs change is released.~~ all logic is on df side now --------- Co-authored-by: Claude Opus 4.6 <[email protected]>
1 parent 7b24735 commit 3f501f4

10 files changed

Lines changed: 557 additions & 90 deletions

File tree

‎datafusion/datasource-parquet/src/access_plan.rs‎

Lines changed: 109 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,12 @@ use parquet::file::metadata::{ParquetMetaData, RowGroupMetaData};
9191
pub struct ParquetAccessPlan {
9292
/// How to access the i-th row group
9393
row_groups: Vec<RowGroupAccess>,
94+
/// Whether all rows in the i-th row group are known to match the predicate.
95+
///
96+
/// This is tracked separately from [`RowGroupAccess`] because it describes
97+
/// whether row-level filter evaluation can be skipped, not which rows should
98+
/// be read.
99+
fully_matched: Vec<bool>,
94100
}
95101

96102
/// Describes how the parquet reader will access a row group
@@ -104,6 +110,24 @@ pub enum RowGroupAccess {
104110
Selection(RowSelection),
105111
}
106112

113+
/// A consecutive set of row groups that share the same row filter requirement.
114+
#[derive(Debug, Clone, PartialEq)]
115+
pub(crate) struct RowGroupRun {
116+
/// True if this run needs row filter evaluation.
117+
pub(crate) needs_filter: bool,
118+
/// The access plan for this run.
119+
pub(crate) access_plan: ParquetAccessPlan,
120+
}
121+
122+
impl RowGroupRun {
123+
fn new(needs_filter: bool, access_plan: ParquetAccessPlan) -> Self {
124+
Self {
125+
needs_filter,
126+
access_plan,
127+
}
128+
}
129+
}
130+
107131
impl RowGroupAccess {
108132
/// Return true if this row group should be scanned
109133
pub fn should_scan(&self) -> bool {
@@ -119,24 +143,34 @@ impl ParquetAccessPlan {
119143
pub fn new_all(row_group_count: usize) -> Self {
120144
Self {
121145
row_groups: vec![RowGroupAccess::Scan; row_group_count],
146+
fully_matched: vec![false; row_group_count],
122147
}
123148
}
124149

125150
/// Create a new `ParquetAccessPlan` that scans no row groups
126151
pub fn new_none(row_group_count: usize) -> Self {
127152
Self {
128153
row_groups: vec![RowGroupAccess::Skip; row_group_count],
154+
fully_matched: vec![false; row_group_count],
129155
}
130156
}
131157

132158
/// Create a new `ParquetAccessPlan` from the specified [`RowGroupAccess`]es
133159
pub fn new(row_groups: Vec<RowGroupAccess>) -> Self {
134-
Self { row_groups }
160+
let row_group_count = row_groups.len();
161+
Self {
162+
row_groups,
163+
fully_matched: vec![false; row_group_count],
164+
}
135165
}
136166

137167
/// Set the i-th row group to the specified [`RowGroupAccess`]
138168
pub fn set(&mut self, idx: usize, access: RowGroupAccess) {
169+
let should_scan = access.should_scan();
139170
self.row_groups[idx] = access;
171+
if !should_scan {
172+
self.fully_matched[idx] = false;
173+
}
140174
}
141175

142176
/// skips the i-th row group (should not be scanned)
@@ -154,6 +188,32 @@ impl ParquetAccessPlan {
154188
self.row_groups[idx].should_scan()
155189
}
156190

191+
/// Marks the i-th row group as fully matched.
192+
///
193+
/// Fully matched row groups are still read according to their
194+
/// [`RowGroupAccess`], but row-level filter evaluation can be skipped.
195+
pub(crate) fn mark_fully_matched(&mut self, idx: usize) {
196+
if self.should_scan(idx) {
197+
self.fully_matched[idx] = true;
198+
}
199+
}
200+
201+
/// Return true if the i-th row group is fully matched and scanned.
202+
pub(crate) fn is_fully_matched(&self, idx: usize) -> bool {
203+
self.should_scan(idx) && self.fully_matched[idx]
204+
}
205+
206+
/// Returns the fully matched row group flags.
207+
pub(crate) fn fully_matched(&self) -> &Vec<bool> {
208+
&self.fully_matched
209+
}
210+
211+
/// Return true if any scanned row group is fully matched.
212+
fn has_fully_matched(&self) -> bool {
213+
self.row_group_index_iter()
214+
.any(|idx| self.is_fully_matched(idx))
215+
}
216+
157217
/// Set to scan only the [`RowSelection`] in the specified row group.
158218
///
159219
/// Behavior is different depending on the existing access
@@ -339,6 +399,54 @@ impl ParquetAccessPlan {
339399
self.row_groups
340400
}
341401

402+
/// Split this plan into consecutive row group runs that share the same row
403+
/// filter requirement.
404+
pub(crate) fn split_runs(self, needs_filter: bool) -> Vec<RowGroupRun> {
405+
if !needs_filter || !self.has_fully_matched() {
406+
return vec![RowGroupRun::new(needs_filter, self)];
407+
}
408+
409+
let num_row_groups = self.row_groups.len();
410+
let row_groups = self.row_groups;
411+
let fully_matched = self.fully_matched;
412+
let mut runs: Vec<RowGroupRun> = Vec::new();
413+
414+
for (idx, (access, fully_matched)) in
415+
row_groups.into_iter().zip(fully_matched).enumerate()
416+
{
417+
if !access.should_scan() {
418+
continue;
419+
}
420+
421+
let row_group_needs_filter = !fully_matched;
422+
if let Some(run) = runs
423+
.last_mut()
424+
.filter(|run| run.needs_filter == row_group_needs_filter)
425+
{
426+
run.access_plan.set(idx, access);
427+
if fully_matched {
428+
run.access_plan.mark_fully_matched(idx);
429+
}
430+
} else {
431+
let mut run_plan = ParquetAccessPlan::new_none(num_row_groups);
432+
run_plan.set(idx, access);
433+
if fully_matched {
434+
run_plan.mark_fully_matched(idx);
435+
}
436+
runs.push(RowGroupRun::new(row_group_needs_filter, run_plan));
437+
}
438+
}
439+
440+
if runs.is_empty() {
441+
vec![RowGroupRun::new(
442+
needs_filter,
443+
ParquetAccessPlan::new_none(num_row_groups),
444+
)]
445+
} else {
446+
runs
447+
}
448+
}
449+
342450
/// Prepare this plan and resolve to the final `PreparedAccessPlan`
343451
pub(crate) fn prepare(
344452
self,

‎datafusion/datasource-parquet/src/metrics.rs‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -213,4 +213,28 @@ impl ParquetFileMetrics {
213213
predicate_cache_records,
214214
}
215215
}
216+
217+
/// Record pages whose page-index pruning was skipped because the containing
218+
/// row group was fully matched by row-group statistics.
219+
///
220+
/// The counter is only registered when there is a non-zero value. This keeps
221+
/// [`ParquetFileMetrics::new`] from cloning the filename and metrics set for
222+
/// files that never use this metric.
223+
pub(crate) fn add_page_index_pages_skipped_by_fully_matched(
224+
metrics: &ExecutionPlanMetricsSet,
225+
partition: usize,
226+
filename: &str,
227+
n: usize,
228+
) {
229+
if n == 0 {
230+
return;
231+
}
232+
233+
let count = MetricBuilder::new(metrics)
234+
.with_new_label("filename", filename.to_string())
235+
.with_type(MetricType::Summary)
236+
.with_category(MetricCategory::Rows)
237+
.counter("page_index_pages_skipped_by_fully_matched", partition);
238+
count.add(n);
239+
}
216240
}

0 commit comments

Comments
 (0)