Skip to content

Commit ebe07a1

Browse files
Dandandanclaude
andcommitted
Skip per-row filter evaluation when all row groups are fully matched
When statistics prove that every remaining row group fully satisfies the filter predicate, skip attaching the row filter to the Parquet decoder entirely. This avoids unnecessary per-row filter evaluation for queries like `WHERE col <> 0` or `WHERE col <> ''` when min/max statistics show the filter is trivially true for all row groups. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
1 parent c17c87c commit ebe07a1

1 file changed

Lines changed: 9 additions & 0 deletions

File tree

‎datafusion/datasource-parquet/src/opener.rs‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1048,6 +1048,15 @@ impl RowGroupsPrunedParquetOpen {
10481048
row_groups.prune_by_limit(limit, rg_metadata, &prepared.file_metrics);
10491049
}
10501050

1051+
// If all remaining row groups are fully matched by the predicate
1052+
// (i.e., statistics prove every row satisfies the filter), skip
1053+
// per-row filter evaluation entirely.
1054+
let all_fully_matched = row_filter.is_some()
1055+
&& row_groups
1056+
.row_group_indexes()
1057+
.all(|idx| row_groups.is_fully_matched()[idx]);
1058+
let row_filter = if all_fully_matched { None } else { row_filter };
1059+
10511060
// Page index pruning: if all data on individual pages can
10521061
// be ruled using page metadata, rows from other columns
10531062
// with that range can be skipped as well.

0 commit comments

Comments
 (0)