Repository navigation
Commit 3f501f4
Skip RowFilter and page pruning for fully matched row groups (#21637)
## Which issue does this PR close?
- Closes #19028.
## Rationale for this change
When DataFusion evaluates a Parquet scan with filter pushdown, it uses
row group statistics to determine which row groups contain matching
rows. The `RowGroupAccessPlanFilter` already tracks which row groups are
"fully matched" — where statistics prove that **all** rows satisfy the
predicate (via `is_fully_matched`).
However, this information was not propagated downstream. Even for fully
matched row groups:
1. **Page index pruning** still evaluated page-level statistics (wasted
work since no pages can be pruned)
2. **RowFilter evaluation** still decoded filter columns and evaluated
the predicate for every row (wasted work since every row passes)
This is especially costly when filter columns are expensive to decode
(e.g., large strings) or when predicates are complex. Common real-world
examples include time-range filters where entire row groups fall within
the range, or `WHERE status != 'DELETED'` on data with no deleted rows.
## What changes are included in this PR?
### DataFusion changes (this PR)
1. **`row_group_filter.rs`**: `RowGroupAccessPlanFilter::build()` now
returns `(ParquetAccessPlan, Vec<usize>)` — the access plan plus the
indices of fully matched row groups.
2. **`page_filter.rs`**: `prune_plan_with_page_index()` accepts a
`fully_matched_row_groups` parameter and skips page-level pruning for
those row groups.
3. **`opener.rs`**: Wires fully matched row groups through the pipeline
— passes them to page pruning and to the `ParquetPushDecoderBuilder` via
`with_fully_matched_row_groups()`.
### Arrow-rs dependency (apache/arrow-rs#9694)
The new `ArrowReaderBuilder::with_fully_matched_row_groups()` API in
arrow-rs allows skipping `RowFilter` evaluation during Parquet decoding
for specified row groups. This PR uses `[patch.crates-io]` pointing to
the arrow-rs fork branch until that PR is merged and released.
### Benchmark
Includes a criterion benchmark (`parquet_fully_matched_filter`) using
`ParquetPushDecoder` directly — the same code path DataFusion's async
opener uses. Dataset: 20 row groups × 50K rows, with a 1KB string
payload column and predicate `x < 200` (all row groups fully matched).
| Scenario | Time | vs. baseline |
|----------|------|-------------|
| Filter pushdown, no skip | ~43 ms | baseline |
| **Filter pushdown, with skip** | **~20 ms** | **~2.2x faster** |
| No pushdown at all | ~24 ms | — |
## Are these changes tested?
- All 82 existing non-submodule `datafusion-datasource-parquet` tests
pass (16 failures are pre-existing, caused by missing `parquet-testing`
submodule)
- The benchmark verifies correctness by asserting the expected row count
- Clippy and fmt pass
## Are there any user-facing changes?
No user-facing API changes. This is a transparent performance
optimization — queries that previously worked will now be faster when
row group statistics prove all rows match the predicate.
~~**Note:** This PR depends on apache/arrow-rs#9694. The
`[patch.crates-io]` in `Cargo.toml` will be removed once that arrow-rs
change is released.~~ all logic is on df side now
---------
Co-authored-by: Claude Opus 4.6 <[email protected]>1 parent 7b24735 commit 3f501f4
10 files changed
Lines changed: 557 additions & 90 deletions
File tree
- datafusion
- datasource-parquet/src
- physical-expr-common/src/metrics
- sqllogictest/test_files
- docs/source/user-guide
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
91 | 91 | | |
92 | 92 | | |
93 | 93 | | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
94 | 100 | | |
95 | 101 | | |
96 | 102 | | |
| |||
104 | 110 | | |
105 | 111 | | |
106 | 112 | | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
107 | 131 | | |
108 | 132 | | |
109 | 133 | | |
| |||
119 | 143 | | |
120 | 144 | | |
121 | 145 | | |
| 146 | + | |
122 | 147 | | |
123 | 148 | | |
124 | 149 | | |
125 | 150 | | |
126 | 151 | | |
127 | 152 | | |
128 | 153 | | |
| 154 | + | |
129 | 155 | | |
130 | 156 | | |
131 | 157 | | |
132 | 158 | | |
133 | 159 | | |
134 | | - | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
135 | 165 | | |
136 | 166 | | |
137 | 167 | | |
138 | 168 | | |
| 169 | + | |
139 | 170 | | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
140 | 174 | | |
141 | 175 | | |
142 | 176 | | |
| |||
154 | 188 | | |
155 | 189 | | |
156 | 190 | | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
157 | 217 | | |
158 | 218 | | |
159 | 219 | | |
| |||
339 | 399 | | |
340 | 400 | | |
341 | 401 | | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
342 | 450 | | |
343 | 451 | | |
344 | 452 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
213 | 213 | | |
214 | 214 | | |
215 | 215 | | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
216 | 240 | | |
0 commit comments