You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 20c7daf
Browse filesBrowse the repository at this point in the historyBrowse files
Run the rules that decide requirements before enforcing them
Enforcing requirements first and having the rules that change them repair
afterwards was out of order for joins, not merely slow. A hash join starts as
PartitionMode::Auto, which declares UnspecifiedDistribution for its children
and UnknownPartitioning for its output, so the enforcement pass did nothing
for joins; only once JoinSelection picked a mode did the plan acquire
requirements nothing satisfied, and the repair then re-derived them over the
whole tree, on a tree the first pass had already expanded. Profiling tpch q21
put that second enforce_distribution_requirements pass at 152 us, 72% of the
planning regression. FilterPushdown repaired for a different reason: pushing
a predicate into a source flips its row count from Exact to Inexact, which
changes whether the scan is parallelized, so enforcement wants to see the
statistics as they are after pushdown.
Both rules now run in the analyzer phase ahead of EnforceDistribution, and
neither re-enforces anything. JoinSelection follows the FilterPushdown
pattern: one rule constructed per phase. JoinSelection::new_before_enforcement
is what the default analyzer registers and leaves materialising distribution
to the pass that follows; JoinSelection::new keeps re-establishing distribution
for the joins it changes, so it stays safe to register at any position. Both
still normalise InterleaveExec back to UnionExec, which is what fixes the
panic in apache#21826 and holds in either order. WindowTopN keeps its repair: it
needs the enforced plan to rebuild the window over a hash-partitioned
PartitionedTopKExec, and it genuinely disturbs both distribution and
ordering, so it is the one optimizer rule the contract is for.
Measured on tpch q21, same machine: merge-base 1.2312 ms, this branch as
reviewed 1.3973 ms, now 1.1823 ms. The split into EnforceDistribution and
EnforceSorting is not where the cost was: on main EnsureRequirements already
runs distribution and sorting as two separate bottom-up walks, and the two
phases measured at 2.3 us and 6.6 us against 292 us for distribution itself.
Plan changes across the full sqllogictest suite are now limited to the rule
trace in explain.slt. window_topn.slt keeps the improved plan, and
repartition_scan.slt keeps its four file groups without any repair, since
enforcement now runs after the predicate reaches the source. The thirty
standalone FilterPushdown snapshots lose the RoundRobin repartition the old
repair added; in the real pipeline EnforceDistribution runs right after it.
Review feedback folded in: PR-history comments removed from optimizer.rs,
analyzer.rs and the ensure_requirements test header, and the EnforceSorting
docs now say its PhysicalOptimizerRule impl is only a compatibility shim.
| 1 |`aggregate_statistics`| - | Uses exact source statistics to answer some aggregates without scanning data. |
138
-
| 2 |`join_selection`| - | Chooses join implementation, build side, and partition mode; re-establishes distribution validity for joins it changes. |
139
-
| 3 |`LimitedDistinctAggregation`| - | Pushes limit hints into grouped distinct-style aggregations when only a small result is needed. |
140
-
| 4 |`FilterPushdown`| pre-optimization phase | Pushes supported physical filters down toward data sources; re-establishes validity for what it changes. |
141
-
| 5 |`WindowTopN`| - | Replaces eligible row-number window and filter patterns with per-partition TopK execution; re-enforces validity. |
0 commit comments