Repository navigation
Commit 5a8ca02
committed
perf(parquet-scan): skip dynamic filters in pushdown=false post-scan filter
The pushdown=false path in the parquet opener split the whole predicate
into 'post_scan_conjuncts' — a per-batch FilterExec-equivalent — which
included any dynamic filter conjuncts (HashJoin bounds, TopK threshold,
aggregate dynamic filter).
For join-heavy TPC-H / TPC-DS this dominates cost: HashJoin's Partitioned-
mode dynamic filter is a 'CASE hash(col) % N WHEN pid THEN bounds ELSE
lit(false) END' — per-row hash + modulo + CASE branch — and it prunes
almost nothing on high-match-rate joins where the downstream hash lookup
would eliminate the same rows anyway. Local TPC-H SF1 Q9 profile showed
1.1% self-time in 'expressions::case::PartialResultIndex::merge_n' and
1.3% in 'arrow_select::filter::filter_native' on the PR, both at 0% on
main — driving Q9 from 40ms → 80ms (2.09x on CI, 1.79x locally).
This commit filters DynamicFilterPhysicalExpr-containing conjuncts out
of 'post_scan_conjuncts'. Effects:
- RowGroupPruner (added by #22450) still sees the full predicate via
prepared.predicate, so RG-level dynamic pruning continues to fire on
bounds/threshold updates.
- pushdown_filters=true path unchanged — dynamic filters still go
through the arrow-rs RowFilter.
- Downstream operator does the exact equivalent: HashJoin's hash lookup
filters rows the bounds would have filtered; TopK's sort heap filters
rows the threshold would have filtered. No wrong results.
Local TPC-H SF1 (release-nonlto, 3 iters):
- baseline (HEAD~2, pre-#22384) avg: 27.65 ms
- PR + this fix avg: 26.24 ms (net 5% ahead of baseline)
- Q9 individually: 34.54 → 34.33 ms (matches baseline, was 80.74 before)
Also regenerates push_down_filter_parquet.slt for the membership-off
default (from the earlier 'split membership from bounds' commit).1 parent 82ea18c commit 5a8ca02
2 files changed
Lines changed: 25 additions & 14 deletions
File tree
- datafusion
- datasource-parquet/src/opener
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1407 | 1407 | | |
1408 | 1408 | | |
1409 | 1409 | | |
1410 | | - | |
1411 | | - | |
| 1410 | + | |
| 1411 | + | |
| 1412 | + | |
| 1413 | + | |
| 1414 | + | |
| 1415 | + | |
| 1416 | + | |
| 1417 | + | |
| 1418 | + | |
1412 | 1419 | | |
1413 | 1420 | | |
| 1421 | + | |
| 1422 | + | |
| 1423 | + | |
1414 | 1424 | | |
1415 | 1425 | | |
1416 | 1426 | | |
| |||
0 commit comments