Repository navigation
Commit cae5fe6
hash join: wrap pushed-down dynamic filter in OptionalFilterPhysicalExpr
The adaptive parquet scheduler in `selectivity::demote_or_drop` only
drops a filter (`FilterState::Dropped`) when its expression is wrapped
in `OptionalFilterPhysicalExpr`; otherwise the filter can only ever be
demoted to post-scan, where it then runs on every batch forever.
Hash-join dynamic filters had no wrapper, so for join-heavy queries
with low-selectivity build sides (most of TPC-H) they cycled between
row-level and post-scan and never got dropped. On SF1 with
`pushdown_filters=true` this dominated runtime — Q9 was 121.5 ms vs
68.5 ms on main (1.77x), Q17 134 ms → 134 ms, etc. Disabling join
dynamic filter pushdown entirely brought Q9 to 49 ms, isolating the
cost to "running ineffective dynamic filters that never get dropped".
Wrap the `Arc<DynamicFilterPhysicalExpr>` in `OptionalFilterPhysicalExpr`
at the `gather_filters_for_pushdown` handoff to mark it as
drop-eligible, and unwrap it in `handle_child_pushdown_result` before
`Arc::downcast::<DynamicFilterPhysicalExpr>` so the join keeps the bare
filter for `.update()` / `.mark_complete()`. Join correctness does not
depend on this filter — it's a probe-side scan-time hint — so dropping
it is always safe.
TPC-H SF1, pushdown=true (mean of 8 iters, drop iter 0), branch / main:
Q3 0.74x Q5 0.73x Q7 0.74x Q8 0.88x
Q9 0.80x Q14 0.69x Q17 0.55x Q20 0.89x
All 963 hash-join unit tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>1 parent ada0472 commit cae5fe6
1 file changed
Lines changed: 23 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
90 | 90 | | |
91 | 91 | | |
92 | 92 | | |
93 | | - | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
94 | 96 | | |
95 | 97 | | |
96 | 98 | | |
| |||
1644 | 1646 | | |
1645 | 1647 | | |
1646 | 1648 | | |
1647 | | - | |
| 1649 | + | |
| 1650 | + | |
| 1651 | + | |
| 1652 | + | |
| 1653 | + | |
| 1654 | + | |
1648 | 1655 | | |
1649 | | - | |
| 1656 | + | |
| 1657 | + | |
| 1658 | + | |
1650 | 1659 | | |
1651 | 1660 | | |
1652 | 1661 | | |
| |||
1667 | 1676 | | |
1668 | 1677 | | |
1669 | 1678 | | |
1670 | | - | |
| 1679 | + | |
| 1680 | + | |
| 1681 | + | |
| 1682 | + | |
| 1683 | + | |
| 1684 | + | |
| 1685 | + | |
| 1686 | + | |
| 1687 | + | |
| 1688 | + | |
| 1689 | + | |
1671 | 1690 | | |
1672 | 1691 | | |
1673 | 1692 | | |
| |||
0 commit comments