You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c856062
Browse filesBrowse the repository at this point in the historyBrowse files
perf(parquet): adaptive selectivity gate for PostScanFilter (MVP)
Adds a per-file adaptive skip to PostScanFilter (Layer 3, the
pushdown_filters=false row-level path introduced by apache#22384). After
observing POST_SCAN_FILTER_SAMPLE_ROWS (65536) rows, if the filter
has pruned fewer than POST_SCAN_FILTER_MIN_SELECTIVITY (2%) of them,
it permanently disables itself for the rest of the file — subsequent
batches take a batch-clone fast path.
Rationale: on the hj benchmark, the density=1 pattern (Q1 Inner,
Q16-18/Q22 RightSemi, Q23 Inner high-fanout) shows Layer 3 running
its predicate on tens of millions of probe rows only to prune zero
— pure per-row tax with no downstream savings, because the
downstream HashJoin hash lookup is going to do the exact match on
the same rows anyway. Adaptive skip detects this after two decoded
batches and stops paying the tax; the downstream operator is
unchanged and produces the same result.
Correctness: PostScanFilter is a semantics-preserving optimization
(the downstream operator does the exact filtering), so disabling
it never produces wrong results — the filter never NEEDED to run,
it was only there to save downstream work.
Scope: this is a deliberately-small per-file MVP. @adriangb's apache#22883
EPIC (Adaptive predicate evaluation) has a full framework with
OptionalFilterPhysicalExpr + a cross-scan SelectivityTracker; this
commit can be superseded by that when it lands.
Local TPC-H SF1 (release-nonlto, 3 iters, avg ms):
- baseline (main, pre-apache#22384): 27.65
- PR + union bounds only: 26.99
- PR + this adaptive skip: 25.98 (net -6% vs baseline)
- Q1 individually: 56.01 (baseline) -> 44.30 (this commit),
i.e. 21% faster than baseline where Layer 3 previously added tax
Tests:
- adaptive_disables_when_selectivity_is_zero
- adaptive_stays_enabled_when_selectivity_is_high
- adaptive_does_not_disable_before_sample_threshold
- adaptive_disable_short_circuits_evaluation
Also reverts the RightSemi/RightAnti gate in
HashJoinExec::allow_join_dynamic_filter_pushdown — adaptive skip
subsumes it (a density=1 Semi/Anti probe scan trips the gate the
same way an Inner one does).
0 commit comments